A Robust Algorithm for Emoji Detection in Smartphone Screenshot Images

Bilal Mohammed Bataineh, Mohd Khaled Yousef Shambour


The increasing use of smartphones and social media apps for communication results in a massive number of screenshot images. These images enrich the written language through text and emojis. In this regard, several studies in the image analysis field have considered text. However, they ignored the use of emojis. In this study, a robust two-stage algorithm for detecting emojis in screenshot images is proposed. The first stage localizes the regions of candidate emojis by using the proposed RGB-channel analysis method followed by a connected component method with a set of proposed rules. In the second verification stage, each of the emojis and non-emojis are classified by using proposed features with a decision tree classifier. Experiments were conducted to evaluate each stage independently and assess the performance of the proposed algorithm completely by using a self-collected dataset. The results showed that the proposed RGB-channel analysis method achieved better performance than the Niblack and Sauvola methods. Moreover, the proposed feature extraction method with decision tree classifier achieved more satisfactory performance than the LBP feature extraction method with all Bayesian network, perceptron neural network, and decision table rules. Overall, the proposed algorithm exhibited high efficiency in detecting emojis in screenshot images.


digital images; emoji; recognition; screenshots; text

Full Text:



Chairunnisa, S. & Benedictus, A., Analysis of Emoji and Emoticon Usage in Interpersonal Communication of Blackberry Messenger and WhatsApp Application User, International Journal of Social Sciences and Management, 4(2), pp. 120-126, 2017.

Zhang, D., Jiang, J., Chen, J., Zhang, Q., Lu, Y., Yao, Y. & Li, S., Logan Liu G. & Liu, Q., Smartphone-Based Portable Biosensing System Using Impedance Measurement with Printed Electrodes for 2, 4, 6-Trinitrotoluene (TNT) Detection, Biosensors and Bioelectronics, 70, pp. 81-88, 2015.

Roy, S.D., Bhardwaj, K., Garg, R. & Chaudhury, S., Camera-Based Document Image Matching Using Multi-Feature Probabilistic Information Fusion, Pattern Recognition Letters, 58, pp. 42-50, 2015.

Chiatti, A., Cho, M.J., Gagneja, A. & Yang, X., Text Extraction and Retrieval from Smartphone Screenshots: Building a Repository for Life in Media, arXiv preprint arXiv:1801.01316, 2018.

Chiatti, A., Cho, M.J., Gagneja, A. & Yang, X., Text Extraction from Smartphone Screenshots to Archive in situ Media Behavior, in Proceedings of the Knowledge Capture Conference, ACM: Austin, TX, USA. pp. 1-4, 2017

Barbieri, F., Ballesteros, M. & Saggion, H., Are Emojis Predictable? arXiv preprint arXiv:1702.07285, 2017.

Chang, W-L., The Power of Emoticon in Social Media, 2017.

Tang, Y. & Hew, K.F., Emoticon, Emoji, and Sticker Use in Computer-Mediated Communications: Understanding Its Communicative Function, Impact, User Behavior, and Motive, in New Media for Educational Change, Springer, pp. 191-201, 2018

Dimson, T., Emojineering Part 1: Machine Learning for Emoji Trends, Instagram Engineering Blog, 30, 2015.

Cappallo, S., Svetlichnaya, S., Garrigues, P., Mensink, T. & Snoek, C.G.M., The New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval, arXiv preprint arXiv:1801.10253, 2018.

Chen, D., Ren, S., Wei, Y., Cao, X. & Sun, J., Joint Cascade Face Detection and Alignment, in European Conference on Computer Vision, Springer, 2014

Ye, Q. & Doermann, D., Text Detection and Recognition In Imagery: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(7), pp. 1480-1500, 2015.

Chavre, P.B. & Ghotkar, A., A Survey on Text Localization Method in Natural Scene Image, International Journal of Computer Applications, 112(13), 2015.

Zafeiriou, S., Zhang, C. & Zhang, Z., A Survey on Face Detection in the Wild: Past, Present and Future, Computer Vision and Image Understanding, 138, pp. 1-24, 2015.

Song, S. & Xiao, J., Deep Sliding Shapes for Amodal 3d Object Detection in RGB-D Images, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.

Cheng, G. & Han, J., A Survey on Object Detection in Optical Remote Sensing Images, ISPRS Journal of Photogrammetry and Remote Sensing, 117, pp. 11-28, 2016

Nguyen, D.T., Li, W. & Ogunbona, P.O., Human Detection from Images and Videos: A Survey, Pattern Recognition, 51, pp. 148-175, 2016.

Walther, J.B. & D’Addario, K.P., The Impacts of Emoticons on Message Interpretation in Computer-Mediated Communication, Social science computer review, 19(3), pp. 324-347, 2001.

Miller, H., “Blissfully Happy” or “Ready to Fight”: Varying Interpretations of Emoji, Proceedings of ICWSM, 2016. 2016.

Kelly, R. & Watts, L., Characterising the Inventive Appropriation of Emoji as Relationally Meaningful in Mediated Close Personal Relationships, Experiences of Technology Appropriation: Unanticipated Users, Usage, Circumstances, and Design, 2015.

Cappallo, S., Mensink, T. & Snoek, C.G., Query-By-Emoji Video Search. in Proceedings of the 23rd ACM International Conference On Multimedia, ACM, 2015.

Felbo, B., Using Millions of Emoji Occurrences to Learn Any-Domain Representations for Detecting Sentiment, Emotion and Sarcasm, arXiv preprint arXiv:1708.00524, 2017.

El Ali, A., Face2emoji: Using Facial Emotional Expressions to Filter Emojis, in Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, ACM, 2017.

Li, X., Yan, R. & Zhang, M., Joint Emoji Classification and Embedding Learning, in Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint Conference on Web and Big Data, Springer, 2017.

Esser, D., Muthmann, K., & Schuster, D., Information Extraction Efficiency of Business Documents Captured with Smartphones and Tablets, in Proceedings of The 2013 ACM Symposium on Document Engineering, ACM: Florence, Italy, pp. 111-114, 2013.

Seeri, S.V., Pujari, J. & Hiremath, P., Text Localization and Character Extraction in Natural Scene Images Using Contourlet Transform and SVM Classifier, International Journal of Image, Graphics and Signal Processing, 8(5), p. 36, 2016.

Simon, C. & Park, I.K., Correcting Geometric and Photometric Distortion of Document Images on a Smartphone, Journal of Electronic Imaging, 24(1), pp. 013038, 2015.

Snoussi, S. & Wahabi, Y., Arabic Document Segmentation on a Smartphone towards Big Data HAJJ rules Extraction, in 1st International Workshop on Arabic Script Analysis and Recognition (ASAR), IEEE, 2017.

Belhedi, A. & Marcotegui, B., Adaptive Scene-Text Binarisation On Images Captured by Smartphone, IET Image Processing, 10(7), pp. 515-523, 2016.

Kumar, J., Ye, P. & Doermann, D., A Dataset for Quality Assessment of Camera Captured Document Images, in International Workshop on Camera-Based Document Analysis and Recognition, Springer, 2013.

Leal, L.R. & Bezerra, B.L., Smartphone camera document detection via Geodesic Object Proposals, in IEEE Latin American Conference on Computational Intelligence (LA-CCI), IEEE, 2016.

El Bahi, H. & Zatni, A., Text Recognition in Document Images Obtained by a Smartphone Based on Deep Convolutional and Recurrent Neural Network, Multimedia Tools and Applications, pp. 1-29, 2019.

Sophea, P., Text-zone Detection and Rectification in Document Images Captured by Smartphone, First EAI International Conference on Computer Science and Engineering, KL, pp. 1-10, 2017.

Bastida, J.O., Gallego, A.J. & Pertusa, A., Multimodal Object Recognition Using Deep Learning Representations Extracted from Images and Smartphone Sensors, in Iberoamerican Congress on Pattern Recognition, Springer, 2018.

Lu, S., Chen, T., Tian, S., Lim, J.H., & Tan, C.L. Scene Text Extraction Based on Edges and Support Vector Regression. International Journal on Document Analysis and Recognition (IJDAR), 18(2), pp. 125-135, 2015.

Sun, L., A Robust Approach for Text Detection from Natural Scene Images, Pattern Recognition, 48(9), pp. 2906-2920, 2015.

Rajan, V. & Raj, S., Text Detection and Character Extraction in Natural Scene Images Using Fractional Poisson Model, IEEE International Conference in Computing Methodologies and Communication (ICCMC), 2017.

Tian, C., Natural Scene Text Detection with MC–MR Candidate Extraction and Coarse-To-Fine Filtering, Neurocomputing, 260, pp. 112-122, 2017.

Bui, D.T., Landslide Susceptibility Mapping along the National Road 32 of Vietnam Using GIS-based J48 Decision Tree Classifier and Its Ensembles, in Cartography from Pole to Pole, Springer, pp. 303-317, 2014.

Mandloi, G., A Survey on Feature Extraction Techniques for Color Images, International Journal of Computer Science and Information Technologies, 5(3), pp. 4615-4620, 2014.

Bui, D.T., A Comparative Assessment Between the Application of Fuzzy Unordered Rules Induction Algorithm and J48 Decision Tree Models in Spatial Prediction of Shallow Landslides at Lang Son City, Vietnam, in Remote Sensing Applications in Environmental Research, Springer, pp. 87-111, 2014.

Zhao, Y. & Zhang, Y., Comparison of Decision Tree Methods for Finding Active Objects, Advances in Space Research, 41(12), pp. 1955-1959, 2008.

Bhargava, N., Decision Tree Analysis on J48 Algorithm for Data Mining, Proceedings of International Journal of Advanced Research in Computer Science and Software Engineering, 3(6), pp. 1114-1119, 2013.

Ludwig, S.A., Picek, S. & Jakobovic, D., Classification of Cancer Data: Analyzing Gene Expression Data Using a Fuzzy Decision Tree Algorithm, in Operations Research Applications in Health Care Management, Springer, pp. 327-347, 2018.

Schomakers, E.M., Internet Users’ Perceptions of Information Sensitivity–Insights from Germany, International Journal of Information Management, 46, pp. 142-150, 2019.

Wang, N., Xu, H. & Grossklags, J., Third-Party Apps on Facebook: Privacy and the Illusion of Control, In Proceedings of The 5th ACM Symposium on Computer Human Interaction for Management of Information Technology, ACM, 2011.

Zhang, J. & Kasturi, R., Extraction of Text Objects in Video Documents: Recent Progress, in Document Analysis Systems, The Eighth IAPR International Workshop on, IEEE, 2008.

Jung, K., Kim, K.I. & Jain, A.K., Text Information Extraction in Images and Video: A Survey, Pattern recognition, 37(5), pp. 977-997, 2004.

Zhiwei, Z., Linlin, L. & Lim, T.C., Edge-based Binarization for Video Text Images, in Pattern Recognition (ICPR), 2010 20th International Conference on, IEEE, 2010.

Lyu, M.R., Song, J. & Cai, M., A Comprehensive Method for Multilingual Video Text Detection, Localization, and Extraction, IEEE Transactions on Circuits and Systems for Video Technology, 15(2), pp. 243-255, 2005.

Pratikakis, I., Gatos, B. & Ntirogiannis, K., H-DIBCO 2010-handwritten Document Image Binarization Competition, in 2010 12th International Conference on Frontiers in Handwriting Recognition, IEEE, 2010.

Bataineh, B., Abdullah, S.N.H.S. & Omar, K., An Adaptive Local Binarization Method for Document Images Based on a Novel Thresholding Method and Dynamic Windows, Pattern Recognition Letters, 32(14), pp. 1805-1813, 2011.

Khurshid, K., Comparison of Niblack Inspired Binarization Methods for Ancient Documents, in Document Recognition and Retrieval XVI, International Society for Optics and Photonics, 2009.

Sauvola, J. & Pietikäinen, M., Adaptive Document Image Binarization, Pattern recognition, 33(2), pp. 225-236, 2000.

Camlica, Z., Tizhoosh, H.R. & Khalvati, F., Medical Image Classification via SVM Using LBP Features from Saliency-based Folded Data, in Machine Learning and Applications (ICMLA), 2015 IEEE 14th International Conference on, IEEE, 2015.

Brewster, E., Keller, J. & Popescu, M., A New Approach for Extracting Texture Features to Aid Detection of Explosive Hazards Using Synthetic Aperture Acoustic Sensing, in Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XXII, International Society for Optics and Photonics, 2017.

Li, W., Local Binary Patterns and Extreme Learning Machine for Hyperspectral Imagery Classification, IEEE Trans. Geoscience and Remote Sensing, 53(7), pp. 3681-3693, 2015.

Liu, L., Local Binary Features for Texture Classification: Taxonomy and Experimental Study, Pattern Recognition, 62, pp. 135-160, 2017.

DOI: http://dx.doi.org/10.5614%2Fitbj.ict.res.appl.2019.13.3.2


  • There are currently no refbacks.

Contact Information:

ITB Journal Publisher, LPPM – ITB, 

Center for Research and Community Services (CRCS) Building Floor 7th, 
Jl. Ganesha No. 10 Bandung 40132, Indonesia,

Tel. +62-22-86010080,

Fax.: +62-22-86010051;

e-mail: jictra@lppm.itb.ac.id.