Translating SIBI (Sign System for Indonesian Gesture) Gesture-to-Text in Real-Time using a Mobile Device

Misael Jonathan; Erdefi Rakun

doi:10.5614/itbj.ict.res.appl.2022.16.3.5

Authors

Misael Jonathan Faculty of Computer Science, University of Indonesia, Kampus Baru UI, Depok 16424, Indonesia
Erdefi Rakun Faculty of Computer Science, University of Indonesia, Kampus Baru UI, Depok 16424, Indonesia

DOI:

https://doi.org/10.5614/itbj.ict.res.appl.2022.16.3.5

Keywords:

Android, gesture-to-text translation, Indonesian sign language recognition, TensorFlow, mobile application, on-device inference

Abstract

The SIBI gesture translation framework by Rakun was built using a series of machine learning technologies: MobileNetV2 for feature extraction, Conditional Random Field for finding the epenthesis movement frame, and Long Short-Term Memory for word classification. This high computational translation system was previously implemented on a personal computer system, which lacks portability and accessibility. This study implemented the system on a smartphone using an on-device inference method: the translation process is embedded into the smartphone to provide lower latency and zero data usage. The system was then improved using a parallel multi-inference method, which reduced the average translation time by 25%. The final mobile SIBI gesture-to-text translation system achieved a word accuracy of 90.560%, a sentence accuracy of 64%, and an average translation time of 20 seconds.

Downloads

Download data is not yet available.

Author Biography

Misael Jonathan, Faculty of Computer Science, University of Indonesia, Kampus Baru UI, Depok 16424, Indonesia

Misael Jonathan received his bachelor degree from Faculty of Computer Science, the University of Indonesia, in 2020. He is currently a software engineer in a local company in Indonesia, specialize in Android application development. He has interests in modern mobile application development and already studying this field since 2019.

References

Siswomartono, S., Simple Way to Learn SIBI (Sign System for Indonesian Gesture), Jakarta: Indonesian National Federation of Welfare for the Deaf, 2007.

Rakun, E., Arymurthy, A.M., Stefanus, L.Y., Wicaksono, A.F. & Wisesa, I.W.W., Recognition of Sign Language System for Indonesian Language using Long Short-Term Memory Neural Networks, Adv. Sci. Lett., 4(2), pp. 400-407, 2016.

Rakun, E., SIBI Gesture to Text Translation System on a Mobile Cellular Device, Indonesia Patent No. IDP000078068, 2021.

Harits, M., Rakun, E. & Hardianto, D., Feature Extraction from Smartphone Images by Using Elliptical Fourier Descriptor, Centroid and Area for Recognizing Indonesian Sign Language SIBI (Sistem Isyarat Bahasa Indonesia), in 2nd International Conference on Intelligent Autonomous Systems, 2019.

Shaik, K.B., Ganesan, P., Kalist, V., Sathish, B. & Jenitha, J.M.M., Comparative Study of Skin Color Detection and Segmentation in HSV and YCbCr Color Space, Procedia Comput. Sci., 57, pp. 41-48, 2015.

Viola, P. & Jones, M., Rapid Object Detection Using A Boosted Cascade of Simple Features, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001.

Kuhl, F.P. & Giardina, C.R., Elliptic Fourier Features of a Closed Contour, Comput. Graph. Image Process., 18(3), pp. 236?258, 1982.

Hochreiter, S. and Schmidhuber, J., Long Short-Term Memory, Neural Comput., 9(8), pp. 1735?1780, 1997.

Pratama, A., Rakun, E. & Hardianto, D., Human Skeleton Feature Extraction from 2-Dimensional Video of Indonesian Language Sign System (SIBI [Sistem Isyarat Bahasa Indonesia]) Gestures, in International Conference on Computing and Artificial Intelligence, 2019.

Tanibata, N., Shimada, N. & Shirai, Y., Extraction of Hand Features for Recognition of Sign Language, Int. Conf. Vis. Interface, pp. 391-398, 2002.

Lucas, B.D. & Kanade, T., An Iterative Image Registration Technique with an Application to Stereo Vision, in International Joint Conference on Artificial Intelligence, 1981.

Anggraini, K., Rakun, E. & Stefanus, L.Y., Recognizing the Components of Inflectional Word Gestures in Indonesian Sign System Known as SIBI (Sistem Isyarat Bahasa Indonesia) by using Lip Motion, in International Conference on Electrical Engineering and Informatics (ICEEI), 2019.

Sagonas, C., Tzimiropoulos, G., Zafeiriou, S. & Pantic, M., 300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge, in IEEE International Conference on Computer Vision Workshops, 2013.

Assael, Y.M., Shillingford, B., Whiteson, S. & Freitas, N.D., LipNet: End-to-End Sentence-level Lipreading, arXiv Learn., 2017.

Setyono, N. & Rakun, E., Recognizing Word Gesture in Sign System for Indonesian Language (SIBI) Sentences Using DeepCNN and BiLSTM, in International Conference on Advanced Computer Science and Information Systems (ICACSIS), 2019.

He, K., Zhang, X., Ren, S. & Sun, J., Deep Residual Learning for Image Recognition, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L.C., MobileNetV2: Inverted Residuals and Linear Bottlenecks, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018.

Maulina, N. & Rakun, E., Recognizing Finger spelling in SIBI (Sistem Isyarat Bahasa Indonesia) using OpenPose and Elliptical Fourier Descriptor, in International Conference on Advanced Information Science and System, 2019.

Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E. & Sheikh, Y., OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields, IEEE Trans. Pattern Anal. Mach. Intell.,43(1), pp. 172-186, 2021.

Rakun, E. & Setyono, N., Improving Recognition of SIBI (Sign System for Indonesian Language) Word Gesture Performance by Combining Skeleton and Handshape Features, Manuscr. Submitt. Publ., 2021.

Shoalihin, R. & Rakun, E., Audio Feature Extraction on SIBI Dataset for Speech Recognition, in International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), 2021.

Baroi, O.L., Kabir, M.S.A., Niaz, A., Rakib, A.M., Islam, M.J. & Rahimi, M.J., Effects of Different coefficients on MFCC and PLP for Bangla Speech Corpus using Tied-state Triphone Model, in International Conference on Electrical, Computer and Communication Engineering (ECCE), pp. 1-6, 2019.

Aulia, A., Rakun, E. & Hardianto, D., Human Skeleton Feature Extraction from 2-Dimensional Video of Indonesian Language Sign System (SIBI [Sistem Isyarat Bahasa Indonesia]) Gestureso Title, in ACM Conference Proceedings, 2019.

Rabiner, L.R., A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proc. IEEE, 77, pp. 257-286, 1989.

Halim, K. & Rakun, E., Sign Language System for Bahasa Indonesia (Known as SIBI) Recognizer using TensorFlow and Long Short-Term Memory, in International Conference on Advanced Computer Science and Information Systems ICACSIS, pp. 403-407, 2018.

Widhinugraha, I. & Rakun, E., Indonesian Language Sign System (SIBI) Recognition Using Threshold Conditional Random Fields, in 8th International Conference on Computing and Pattern Recognition, pp. 380?384, 2019.

Cho, S.S., Yang, H.D. & Lee, S.W., Sign Language Spotting Based on Semi-Markov Conditional Random Field, 2009 Work. Appl. Comput. Vision, WACV 2009, 2009.

Rakun, E., Widhinugraha, I. & Setyono, N., Word Recognition and Automated Epenthesis Removal for Indonesian Sign System Sentence Gestures, Manuscr. Submitt. Publ., 2021