Mapping Public Emotions Regarding the 2025 UTBK Announcement in Indonesia: A Multi-Label Approach with Targeted Calibration
Keywords:
multi-label, emotion, calibration, IndoBERTweet, rate-targeting, lexicon boost, IndonesiaAbstract
This study maps public emotions in Indonesian-language tweets related to the 2025 UTBK announcement using multi-label emotion classification. The main challenges in multi-label emotion classification on social media include extreme label imbalance, distribution shift between training and application data, and weak lexical signals for specific emotions. This study aims to build a reliable emotion modeling framework for long-tail social media corpora while demonstrating generalizable post-training calibration practices. The novelty lies in the integration of four components: (1) per-label posterior calibration using Platt scaling, (2) precision-targeted per-label thresholding frozen from the development set, (3) score-quantile–based rate targeting to align predicted prevalence with domain-based rates, and (4) context-limited lexicon-aware boosting with a final clamp. The proposed pipeline is lightweight and model-agnostic. This research adopts a quantitative experimental approach by varying post-training calibration components to measure their impact on classification performance. An IndoBERTweet model is trained using BCEWithLogitsLoss on manually annotated data, then calibrated and evaluated on development and test sets. The results demonstrate balanced micro- and macro-level performance, improved detection of minority labels, and emotion mapping over 3,500 tweets with prevalence distributions consistent with Plutchik’s theory of emotions.
References
A. Semeraro, S. Vilella, and G. Ruffo, “PyPlutchik: Visualising and comparing emotion-annotated corpora,” PLOS ONE, vol. 16, no. 9, p. e0256503, Sep. 2021. https://doi.org/10.1371/journal.pone.0256503
F. Koto, J. H. Lau, and T. Baldwin, “INDOBERTWEET: A pretrained language model for Indonesian Twitter with effective domain-specific vocabulary initialization,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pp. 10660–10668, Nov. 7–11, 2021. [Online] Available : https://aclanthology.org/2021.emnlp-main.833/
S. Cahyawijaya, H. Lovenia, A. F. Aji, G. I. Winata, B. Wilie, F. Koto, R. Mahendra, et al., “NusaCrowd: open source initiative for Indonesian NLP resources,” Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 13745–13818, 2023. https://doi.org/10.18653/v1/2023.findings-acl.868
P. W. Koh, S. Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al., “WILDS: A benchmark of in-the-wild distribution shifts,” in Proc. 38th Int. Conf. Machine Learning (ICML 2021), PMLR 139, pp. 5637–5664, 2021. [Online] Available : https://proceedings.mlr.press/v139/koh21a/koh21a.pdf
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186, 2019. https://doi.org/10.48550/arXiv.1810.04805
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Machine Learning (ICML 2017), PMLR 70, Sydney, Australia, 2017. [Online] Available : https://proceedings.mlr.press/v70/guo17a/guo17a.pdf
K. Draszawka and J. Szymański, “From scores to predictions in multi-label classification: neural thresholding strategies,” Applied Sciences, vol. 13, no. 13, p. 7591, 2023. https://doi.org/10.3390/app13137591
S. Garg, Y. Wu, S. Balakrishnan, and Z. C. Lipton, “A unified view of label shift estimation,” in Advances in Neural Information Processing Systems (NeurIPS 2020), 2020. [Online] Available : https://proceedings.neurips.cc/paper/2020/
K. Draszawka and J. Szymański, “From scores to predictions in multi-label classification: neural thresholding strategies,” Applied Sciences, vol. 13, no. 13, p. 7591, 2023. https://doi.org/10.3390/app13137591
S. Rajaraman, P. Ganesan, and S. Antani, “Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks,” PLOS ONE, vol. 17, no. 1, p. e0262838, 2022. https://doi.org/10.1371/journal.pone.0262838
R. Riccosan and K. E. Saputra, “Multilabel multiclass sentiment and emotion dataset from Indonesian mobile application review,” Data in Brief, vol. 50, Art. no. 109576, 2023, https://doi.org/10.1016/j.dib.2023.109576
A. K. Glenn, P. LaCasse, and B. Cox, “Emotion classification of Indonesian tweets using bidirectional LSTM,” Neural Computing and Applications, vol. 35, pp. 9567–9578, 2023. https://doi.org/10.1007/s00521-022-08186-1
M. H. Algifari and E. D. Nugroho, “Emotion classification of Indonesian tweets using BERT embedding,” Journal of Applied Informatics and Computing, vol. 7, no. 2, pp. 172–176, 2023. https://doi.org/10.30871/jaic.v7i2.6528
Z. C. Lipton, Y.-X. Wang, and A. J. Smola, “Detecting and correcting for label shift with Black Box Predictors,” in Proc. 35th Int. Conf. Machine Learning (ICML 2018), PMLR 80, Stockholm, Sweden, 2018, [Online] Available : https://proceedings.mlr.press/v80/lipton18a/lipton18a.pdf
C. Wang, “Calibration in Deep Learning: a survey of the state-of-the-art,” Arxiv: 2308.01222, 2025. https://doi.org/10.48550/arXiv.2308.01222
J. W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (4th ed.). Sage Publications, 2014. [Online] Available : https://www.ucg.ac.me/skladiste
JustAnotherArchivist, “Snscrape: A social networking service scraper in Python,” GitHub repository, 2021. [Online]. Available: https://github.com/JustAnotherArchivist/snscrape. Accessed: Sep. 26, 2025.
S. Kishore, D. Sundaram, and M. D. Myers, “A temporal dynamics framework and methodology for computationally intensive social media research,” Journal of Information Technology, vol. 40, no. 2, pp. 140–163, 2025. https://doi.org/10.1177/02683962241283051
L. Bozarth and C. Budak, “Keyword expansion techniques for collecting Twitter data on social movements,” EPJ Data Science, vol. 11, no. 30, 2022. https://doi.org/10.1140/epjds/s13688-022-00343-9
Paula Vicente, “Sampling Twitter users for social science research: evidence from a systematic review of the literature,” Quality & Quantity, vol. 57, no. 6, pp. 5449–5489, 2023. https://doi.org/10.1007/s11135-023-01615-w
S. M. Mohammad and P. D. Turney, “Crowdsourcing a word–emotion association lexicon,” Computational Intelligence, vol. 29, no. 3, pp. 436–465, 2013. https://doi.org/10.1111/j.1467-8640.2012.00460.x
Y. Qi and Z. Shabrina, “Sentiment analysis using Twitter data: a comparative application of lexicon- and machine-learning-based approach,” Social Network Analysis and Mining, vol. 13, no. 31, 2023. https://doi.org/10.1007/s13278-023-01030-x
M. C. Hinojosa Lee, J. Braet, and J. Springael, “Performance metrics for multilabel emotion classification: Comparing micro, macro, and weighted F1-scores,” Applied Sciences, vol. 14, no. 21, p. 9863, Oct. 2024. https://doi.org/10.3390/app14219863
B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proc. 1st Conf. of the Asia-Pacific Chapter of the Association for Computational Linguistics and 10th Int. Joint Conf. on Natural Language Processing (AACL-IJCNLP), Suzhou, China, pp. 843–857, 2020. [Online] Available : https://aclanthology.org/2020.aacl-main.85/
D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: a dataset of fine-grained emotions,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics (ACL 2020),pp. 4040–4054 , Jul. 2020. https://doi.org/10.18653/v1/2020.acl-main.372
Published
How to Cite
Issue
Section
Copyright (c) 2026 Rizki Yustisia Sari, Sabar Sabar, Joni Joni, Gelard Unthirta Pratama, Ronal Ronal, Fadli Hastito
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
An author who publishes in the Jurnal Otomasi Kontrol dan Instrumentasi agrees to the following terms:
- The author retains the copyright and grants the journal the right of first publication of the work simultaneously licensed under the Creative Commons Attribution-ShareAlike 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal
- Author can enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book) with the acknowledgement of its initial publication in this journal.
