Development and Experimental Evaluation of a Smartphone-Based Mixed Reality Gun Simulator
Keywords:
mixed reality, virtual reality, augmented reality, smartphone-based simulation, human-computer interactionAbstract
Mixed Reality (MR) has gained increasing attention in simulation and training because it enables immersive interaction between virtual and physical environments. However, many existing MR systems rely on expensive head-mounted displays and dedicated tracking hardware, limiting their accessibility for educational and proof-of-concept research. This study aims to develop and experimentally evaluate a low-cost smartphone-based MR gun simulator operating in Virtual Reality (VR) and Augmented Reality (AR) modes. Its novelty lies in integrating consumer-grade hardware into a unified MR platform that supports immersive interaction at an affordable cost. The prototype integrates a smartphone, a custom handheld controller, embedded sensing, computer vision, and a real-time simulation environment. Experimental evaluation examined crosshair mapping accuracy, response latency, system frame rate performance, and AR object detection reliability. Results show that the prototype achieved acceptable interaction accuracy, responsive real-time performance, stable rendering in both VR and AR modes, and reliable object detection throughout the evaluation. These findings demonstrate that affordable smartphone-based MR systems can provide sufficient performance for proof-of-concept research and educational applications, establishing a practical foundation for developing immersive training and human-computer interaction systems.
References
S. Picard, N. Sun, and J. Botev, "XR MUSE: An open-source Unity framework for extended reality-based networked multi-user studies," Virtual Worlds, vol. 3, no. 4, pp. 404–417, 2024. https://doi.org/10.3390/virtualworlds3040022
V. De Luca, M. C. Barba, G. D'Errico, B. L. Nuzzo, and L. T. De Paolis, "A user experience analysis for a mobile Mixed Reality application for cultural heritage," Virtual Reality, vol. 27, pp. 2821–2837, 2023. https://doi.org/10.1007/s10055-023-00840-w
T. N. Fitria, "Augmented Reality (AR) and Virtual Reality (VR) technology in education: media of teaching and learning : A review," International Journal of Computer and Information System (IJCIS), vol. 4, no. 1, pp. 14–25, 2023. https://doi.org/10.29040/ijcis.v4i1.102
F. A. Aziz, A. S. M. A. Alsaeed, S. Sulaiman, M. K. A. Mohd Ariffin, and M. F. Al-Hakim, "Mixed Reality improves education and training in assembly processes," Journal of Engineering and Technological Sciences, vol. 52, no. 4, pp. 598–607, 2020. https://doi.org/10.5614/j.eng.technol.sci.2020.52.4.10
J.-S. Kim, K.-W. Kim, H.-J. Kim, and S.-Y. Moon, "Development and usability evaluation of a leap motion-based controller-free VR training system for inferior alveolar nerve block," Applied Sciences, vol. 16, no. 3, Art. no. 1325, 2026. https://doi.org/10.3390/app16031325
D. Ergen and A. N. Yılmaz, "The effects of leap motion-based virtual reality training on performance-based dexterity in children with cerebral palsy: A randomized controlled trial," Egyptian Pediatric Association Gazette, vol. 74, no. 1, Art. no. 49, 2026. https://doi.org/10.1186/s43054-026-00552-1
Y. Lei, Y. Deng, L. Dong, X. Li, X. Li, and Z. Su, "A novel sensor fusion approach for precise hand tracking in virtual reality-based human-computer interaction," Biomimetics, vol. 8, no. 3, Art. no. 326, 2023. https://doi.org/10.3390/biomimetics8030326
M. Gil-Martín, M. R. Marini, R. San-Segundo, and L. Cinque, "Dual leap motion controller 2 : A robust dataset for multi-view hand pose recognition," Scientific Data, vol. 11, Art. no. 1102, 2024. https://doi.org/10.1038/s41597-024-03968-9
A. Houston, V. Walters, T. Corbett, and R. Coppack, "Evaluation of a multi-sensor leap motion setup for biomechanical motion capture of the hand," Journal of Biomechanics, vol. 127, Art. no. 110713, 2021. https://doi.org/10.1016/j.jbiomech.2021.110713
L. Caro, E. Villablanca, and B. Peralta, "A Virtual Reality application using hand motion tracking with leap motion sensor," in 2020 39th International Conference of the Chilean Computer Science Society (SCCC), Coquimbo, Chile, Nov. 2020. https://doi.org/10.1109/SCCC51225.2020.9281210
Y. Zhang and O. E. Meruvia-Pastor, "Operating virtual panels with hand gestures in immersive VR games : Experiences with the leap motion controller," in Augmented Reality, Virtual Reality, and Computer Graphics (Lecture Notes in Computer Science, vol. 10324), L. T. De Paolis and P. Bourdot, Eds. Cham, Switzerland: Springer, 2017, pp. 306–319. https://doi.org/10.1007/978-3-319-60922-5_24
K. Mahdikhanlou and H. Ebrahimnezhad, "Integration of leap motion sensor with camera for better performance in hand tracking," Metaverse, vol. 5, no. 2, Art. no. 3020, 2024. https://doi.org/10.54517/m.v5i2.3020
I. Mostachetti, D. Ferrari, E. Barcella, M. Corna, K. Zavareh, C. Pasini, P. Valentini, and A. Vitali, "Usability Comparison Between Leap Motion Controller and MediaPipe in Serious Games," in dHealth 2026: Proceedings of the 20th Health Informatics Meets Digital Health Conference, Studies in Health Technology and Informatics, vol. 331. Amsterdam, The Netherlands: IOS Press, 2026, pp. 131–136. https://doi.org/10.3233/SHTI260069
Indriani, M. Harris, and A. S. Agoes, "Applying hand gesture recognition for user guide application using MediaPipe," in Proceedings of the 2nd International Seminar of Science and Applied Technology (ISSAT 2021), Advances in Engineering Research, vol. 207. Paris, France: Atlantis Press, 2021, pp. 101–108. https://doi.org/10.2991/aer.k.211106.017
V. Patil, S. Sutar, S. Ghadage, and S. Palkar, "Gesture recognition for media interaction : A streamlit implementation with OpenCV and MediaPipe," International Journal for Research in Applied Science and Engineering Technology, vol. 11, no. 9, pp. 1039–1046, 2023. https://doi.org/10.22214/ijraset.2023.55775
H. Bai, L. Zhang, J. Yang, and M. Billinghurst, "Bringing full-featured mobile phone interaction into Virtual Reality," Computers & Graphics, vol. 97, pp. 42–53, 2021. https://doi.org/10.1016/j.cag.2021.04.004
S. S. Kyian and R. J. Teather, "Selection performance using a smartphone in VR with redirected input," in Proceedings of the ACM Symposium on Spatial User Interaction (SUI '21), Virtual Event, USA, Nov. 9–10, 2021, pp. 1–12. https://doi.org/10.1145/3485279.3485292
L. Schenck, C. Bäumer, B. Ross, G. Schäfer, N. Stember, H. Thomas, S. Stieglitz, and B. Timmermann, "Development of a smartphone virtual reality game to support the radiation therapy of children and adolescents in proton centers," Frontiers in Pediatrics, vol. 11, Art. no. 1163022, 2023. https://doi.org/10.3389/fped.2023.1163022
J.-P. Stauffert, F. Niebling, and M. E. Latoschik, “Latency and cybersickness: Impact, causes, and measures. A review,” Frontiers in Virtual Reality, vol. 1, Art. no. 582204, 2020, https://doi.org/10.3389/frvir.2020.582204.
T. Porcino, D. Reilly, E. Clua, and D. Trevisan, “A guideline proposal for minimizing cybersickness in VR-based serious games and applications,” 2022 , arXiv:2207.06346, https://doi.org/10.48550/arXiv.2207.06346
Tošić, D. Hoffman, and N. Balram, “Effect of latency on simulator sickness in smartphone virtual reality,” Journal of the Society for Information Display, vol. 29, no. 7, pp. 561–572, 2021, https://doi.org/10.1002/jsid.999
T. Bourlai et al., “Development of a human performance baseline of lay error in targeting,” IEEE Access, vol. 13, pp. 91182–91199, 2025, https://doi.org/10.1109/ACCESS.2025.3571591
J. Wang, R. Shi, W. Zheng, W. Xie, D. Kao, and H.-N. Liang, "Effect of frame rate on user experience, performance, and simulator sickness in Virtual Reality," IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2478–2488, May 2023. https://doi.org/10.1109/TVCG.2023.3247057.
Z. Wang, K. Foo, S. Yan, V. A. Gonzalez, and N. Giacaman, “Refresh rate and graphical benchmarks for mobile VR application development,” *Journal of Mobile Multimedia*, vol. 18, no. 6, pp. 1561–1598, 2022, https://doi.org/10.13052/jmm1550-4646.1865
PTC Inc., “Device Performance Optimization,” Vuforia Engine Library. [Online]. Available: https://developer.vuforia.com/library/vuforia-engine/platform-support/camera/device-performance-optimization/. Accessed 12 October 2026
DAQRI, “Motion-to-Photon Latency in Mobile AR and VR,” Medium. [Online]. Available: https://medium.com/@DAQRI/motion-to-photon-latency-in-mobile-ar-and-vr-99f82c480926. Accessed 12 October 2026
Published
How to Cite
Issue
Section
Copyright (c) 2026 Victor Sinaga
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
An author who publishes in the Jurnal Otomasi Kontrol dan Instrumentasi agrees to the following terms:
- The author retains the copyright and grants the journal the right of first publication of the work simultaneously licensed under the Creative Commons Attribution-ShareAlike 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal
- Author can enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book) with the acknowledgement of its initial publication in this journal.
