Assessing The Validity And Reliability Of A Motivation Questionnaire For Programming Learning With Augmented Reality Module: A Pilot Study
DOI:
https://doi.org/10.37134/jictie.vol12.2.2.2025Keywords:
validity and reliability, augmented reality in programming, motivation questionnaire, education, Cronbach Alpha ReliabilityAbstract
Malaysian secondary school students face significant motivational challenges in learning programming, including low self-efficacy, limited immediate feedback, and passive learning environments that hinder intrinsic motivation. To address these issues, Augmented Reality (AR) has been introduced as a potential tool to enhance student motivation in programming education. Although various instruments exist to measure student motivation, such as the Instructional Materials Motivation Survey (IMMS) developed by Keller (2010), there is a lack of studies that specifically employ the IMMS within the context of AR-integrated programming education. Therefore, this study adapted the IMMS for Programming Learning with Augmented namely as Motivation Questionnaire for Programming Learning with Augmented Reality (MQAR). A pilot study was conducted over a period of five weeks to evaluate the validity and reliability of the MQAR. Seven experts were selected to be involved in the content and technological validation, while 30 secondary school students enrolled in a computer science subject were involved in assessing the instrument’s reliability. The data were analysed using experts’ percent agreement and descriptive statistics. The findings revealed that the MQAR demonstrated high content validity (99.26%) and technological validity (99.45%). Interestingly, this study also found that, MQAR are reliable instruments to measure students’ motivation when using Programming Learning with Augmented module with Cronbach’s alpha values indicating good internal consistency across the four motivation constructs: Attention (α = 0.85), Relevance (α = 0.78), Confidence (α = 0.73), and Satisfaction (α = 0.82). The overall reliability of the instrument was α = 0.80, confirming that the MQAR is a valid, reliable, and robust tool for measuring student motivation Questionnaire for Programming Learning with Augmented Reality module. Limitations and future suggestions are acknowledged for this study.
Downloads
References
Abdurrahman. (2019). Developing a STEM learning makerspace for fostering students’ 21st-century skills in the Fourth Industrial Revolution Era. Journal of Physics: Conference Series, 1155, 012002. https://doi.org/10.1088/1742-6596/1155/1/012002
Akcayır, G., & Akcayır, M. (2018). The flipped classroom: A review of its advantages and challenges. Computers & Education, 126, 334–345. https://doi.org/10.1016/j.compedu.2018.07.021
Almdahem, A. (2024). The perceptions of secondary school students regarding the learning of computer programming at Key Stage 4 in the revised curriculum in England: Problems and solutions. International Journal of Computer Science Education in Schools, 7(1). https://doi.org/10.21585/ijcses.v7i1.147
Ankora, C., Bolatimi, S. O., Bensah, L., Mahama, F., Kuadey, N. A., Adu, A. S., & Adjei, L. (2024). Examining students’ academic motivation for studying programming languages. Journal of Computer Assisted Learning, 39(6), 2025–2034. https://doi.org/10.1111/jcal.12862
Aydin, M., & Cakiroglu, U. (2024). Learning programming with an augmented reality‐based editor: A dynamic code visualisation approach. Journal of Computer Assisted Learning, 41(1). https://doi.org/10.1111/jcal.13093
Azman, N. S., Rahmatullah, B., Tamrin, K. F., & Qahtan, Y. M. (2024). A preliminary study on tech-based health advisory online system: The case study of UPSI computing students. AIP Conference Proceedings, 2750, 040006. https://doi.org/10.1063/5.0148931
Bacca, J., Baldiris, S., Fabregat, R., Graf, S., & Kinshuk. (2014). Augmented reality trends in education: A systematic review of research and applications. Journal of Educational Technology & Society, 17(4), 133–149. https://www.jstor.org/stable/jeductechsoci.17.4.133
Bursali, H., & Yilmaz, R. M. (2019). Effect of augmented reality applications on secondary school students’ reading comprehension and learning permanency. Computers in Human Behavior, 95, 126–135. https://doi.org/10.1016/j.chb.2019.01.035
Bybee, R. W. (2013). The Case for STEM Education: Challenges and Opportunities. NSTA Press.
Cardenas, A. I., & Ibarra, V. E. (2022). Examining university students’ satisfaction regarding remote learning during the COVID-19 pandemic. International Journal of Multidisciplinary Research and Growth Evaluation, 573–577. https://doi.org/10.54660/anfo.2022.3.3.29
Cevahir, H., Ozdemir, M., & Baturay, M. H. (2022). The effect of animation-based worked examples supported with augmented reality on the academic achievement, attitude and motivation of students towards learning programming. Participatory Educational Research, 9(3), 226–247. https://doi.org/10.17275/per.22.63.9.3
Che Lah, N. H., Tasir, Z., & Jumaat, N. F. (2021). Applying alternative method to evaluate online problem-solving skill inventory (OPSI) using Rasch model analysis. Educational Studies, 49(4), 644–666. https://doi.org/10.1080/03055698.2021.1874310
Cheah, C. S. (2020). Factors contributing to the difficulties in teaching and learning of Computer Programming: A literature review. Contemporary Educational Technology, 12(2). https://doi.org/10.30935/cedtech/8247
Cheng, K., & Tsai, C. (2020). Students’ motivational beliefs and strategies, perceived immersion and attitudes towards Science Learning with immersive virtual reality: A partial least squares analysis. British Journal of Educational Technology, 51(6), 2140–2159. https://doi.org/10.1111/bjet.12956
Cook, D. A., Beckman, T. J., Thomas, K. G., & Thompson, W. G. (2009). Measuring motivational characteristics of courses: Applying kellerʼs instructional materials motivation survey to a web-based course. Academic Medicine, 84(11), 1505–1509. https://doi.org/10.1097/acm.0b013e3181baf56d
Field, A. (2013). Discovering statistics using IBM SPSS Statistics (4th ed.). Sage Publications.
Fojcik, M., Fojcik, M. K., Høyland, S.-O., & Hoem, J. Ø. (2022). Challenges in teaching programming. Proceedings of the International Conference on Education and New Developments (END 2022), 160–164.
Figueiredo, J., & Garcia-Penalvo, F. J. (2020). Increasing student motivation in computer programming with gamification. 2020 IEEE Global Engineering Education Conference (EDUCON), 997–1000. https://doi.org/10.1109/educon45650.2020.9125283
Feurzeig, W., Papert, S., Bloom, M., Grant, R., & Solomon, C. (1970). Programming-languages as a conceptual framework for teaching mathematics. ACM SIGCUE Outlook, 4(2), 13–17. https://doi.org/10.1145/965754.965757
Garcia, M. A., Homan, P. A., García, C., & Brown, T. H. (2020). The color of COVID-19: Structural racism and the disproportionate impact of the pandemic on older black and Latinx Adults. The Journals of Gerontology: Series B, 76(3). https://doi.org/10.1093/geronb/gbaa114
Ghafar, N. A., Rahmatullah, B., Razak, N. A., Muttallib, F. H. A., Adnan, M. H. M., & Sarah, L. L. (2023). Systematic literature review on digital courseware usage in Geography subjects for secondary school students. Journal of ICT in Education, 10(1), 26-39. https://doi.org/10.37134/jictie.vol10.1.3.2023
Gopalan, V., Zulkifli, A. N., & Bakar, J. A. (2016). A learning performance study between the conventional approach and augmented reality textbook among secondary school students. AIP Conference Proceedings, 1761, 020039. https://doi.org/10.1063/1.4960879
Gunbatar, M. S., & Karalar, H. (2018). Gender differences in middle school students’ attitudes and self-efficacy perceptions towards Mblock programming. European Journal of Educational Research, 7, 925–933. https://doi.org/10.12973/eu-jer.7.4.925
Gul, M. D., & Ayık, Z. (2024). Comprehensive science mapping of stem studies in gifted education. Journal of Turkish Science Education, 21(1), 153–174. https://doi.org/10.36681/tused.2024.009
Hertzog, M. A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191. https://doi.org/10.1002/nur.20247
Ibanez, M. B., Di Serio, Á., Villaran, D., & Delgado Kloos, C. (2014). Experimenting with electromagnetism using augmented reality: Impact on Flow Student Experience and educational effectiveness. Computers & Education, 71, 1–13. https://doi.org/10.1016/j.compedu.2013.09.004
Ivarson, E., Erlandsson, V., Faraon, M., & Khatib, S. (2024). Augmented reality and gamification in Higher Education: Designing Mobile Interaction to enhance students’ motivation and learning. E-Learning and Digital Media. https://doi.org/10.1177/20427530241239981
Jacob, S. R., Tate, T., & Warschauer, M. (2024). Emergent AI-Assisted Discourse: A case study of a second language writer authoring with ChatGPT. Journal of China Computer-Assisted Language Learning. https://doi.org/10.1515/jccall-2024-0011
Julious, S. A. (2005). Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics, 4(4), 287–291. https://doi.org/10.1002/pst.185
Puganesri, & Puteh, S. (2019). Computer Science education in Malaysia schools: The challenges of enhancing computational thinking skills. International Journal of Engineering and Advanced Technology, 8(6s3), 441–444. https://doi.org/10.35940/ijeat.f1080.0986s319
Keller, J. M. (2010). Motivational design for learning and performance. https://doi.org/10.1007/978-1-4419-1250-3
KPM. (2016). Program pengkomputeraan: Sains komputer dalam kurikulum pendidikan. Kementerian Pelajaran Malaysia.
Kesim, M., & Ozarslan, Y. (2012). Augmented reality in education: Current Technologies and the potential for education. Procedia - Social and Behavioral Sciences, 47, 297–302. https://doi.org/10.1016/j.sbspro.2012.06.654
Kishore, O. A. R., Appa Rao, D. Ch., & Bayana, R. K. (2024, February). Unveiling the essence of 21st-century skills in education and the workplace. International Journal of Creative Research Thoughts, 12(2), Article IJCRT2402046.
Latif, A., Jabar, C. S., & Kurniawan, H. L. (2024). A valid and reliable instruments to measure learning motivation of university students in Elementary teacher education. ACP Official Conference Proceedings, 309–319. https://doi.org/10.22492/issn.2187-4743.2024.26
Loorbach, N., Peters, O., Karreman, J., & Steehouder, M. (2014). Validation of the instructional materials motivation survey (IMMS) in a self‐directed instructional setting aimed at working with technology. British Journal of Educational Technology, 46(1), 204–218. https://doi.org/10.1111/bjet.12138
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6). https://doi.org/10.1097/00006199-198611000-00017
Medeiros, M. C., Vasconcelos, G. F., Veiga, Á., & Zilberman, E. (2019). Forecasting inflation in a data-rich environment: The benefits of Machine Learning Methods. Journal of Business & Economic Statistics, 39(1), 98–119. https://doi.org/10.1080/07350015.2019.1637745
Mohajan, H. K. (2017). Two criteria for good measurements in research: Validity and Reliability. Annals of Spiru Haret University. Economic Series, 17(4), 59–82. https://doi.org/10.26458/1746
Mohamad Kasim, F. S., Rahmatullah, B., Rahmatullah Khan, H. K., Rosli, A. N., Che Lah, N. H., Hidayanto, A. N., & Matthíasdóttir, Á. (2023). The effectiveness of online supervision for postgraduate students: A systematic literature review. ASEAN Journal of Teaching & Learning in Higher Education, 15(2), 252-267. https://ejournal.ukm.my/ajtlhe/article/view/65132
Mohamad, N. S., & Siew, N. M. (2022). Learning motivation towards Linear Programming Questionnaire for Polytechnic Diploma Students. International Journal of Education, Psychology and Counseling, 7(47), 364–379. https://doi.org/10.35631/ijepc.747031
Moore, T. J., Stohlmann, M. S., Wang, H.-H., Tank, K. M., Glancy, A. W., & Roehrig, G. H. (2014). Implementation and integration of engineering in K–12 STEM Education. Engineering in Pre-College Settings, 35–60. https://doi.org/10.2307/j.ctt6wq7bh.7
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
Yang, F.-C., Lai, H.-M., & Wang, Y.-W. (2023). Effect of augmented reality-based Virtual Educational Robotics on programming students’ enjoyment of learning, computational thinking skills, and academic achievement. Computers & Education, 195, 104721. https://doi.org/10.1016/j.compedu.2022.104721
Pease, R., Vuke, M., June Maker, C., & Muammar, O. M. (2020). A practical guide for implementing the STEM assessment results in classrooms: Using strength-based reports and real engagement in active problem solving. Journal of Advanced Academics, 31(3), 367–406. https://doi.org/10.1177/1932202x20911643
Pujiastuti, H., & Haryadi, R. (2023). Hybrid learning impact with augmented reality to improve higher order thinking skills of students. International Journal of Advanced and Applied Sciences, 10(12), 7–18. https://doi.org/10.21833/ijaas.2023.12.002
Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming. ACM Transactions on Computing Education, 18(1), 1–24. https://doi.org/10.1145/3077618
Rahmadani, D. F., & Sunarmi, S. (2023). Validity and practicality of the problem-based learning e-module assisted by augmented reality on virus to improve student learning outcomes, retention, and Science Literacy. AIP Conference Proceedings, 2569, 020051. https://doi.org/10.1063/12.0015280
Rahman, N. A. A., Zakaria, N. H., & Din, R. (2021). The integration of new teaching methods must carefully align with national curriculum standards to achieve optimal learning outcomes. 2021 IEEE 13th International Conference on Engineering Education (ICEED), 255–259. https://doi.org/10.1109/ICEED53521.2021.9453041
Raposo, A. C., Lebrilla, C. B., Portela, R. W., Goonatilleke, E., Neto, F. A., & Oriá, A. P. (2020). The proteomics of Roadside Hawk (Rupornis magnirostris), broad-snouted caiman (Caiman latirostris) and Loggerhead Sea Turtle (Caretta Caretta) tears. BMC Veterinary Research, 16(1). https://doi.org/10.1186/s12917-020-02495-0
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037//0003-066x.55.1.68
Saad, A., Khairulanuar, F. N., & Rahmatullah, B. (2023). The development of a mobile application ISMIND for formative students’ assessment: ICT and learning tools in secondary education. Information Technologies and Learning Tools, 98(6), 66–81. https://doi.org/10.33407/itlt.v98i6.5370
Samuri, S. M., Emang, D. A., Agus, R., Rahmatullah, B., Mohamed Salleh, N. S., & Mustafa, M. C. (2021). LACLOD: Learning analytics for children’s logic development. The International Journal of Multimedia & Its Applications, 13. https://aircconline.com/abstract/ijma/v13n2/13221ijma01.html
Silva, L., Gomes, A., Borges, A. R., Vasconcelos, V., & Mendes, A. J. (2023). A study on the motivation of computer science students to learn programming. 2023 International Symposium on Computers in Education (SIIE), 1–7. https://doi.org/10.1109/siie59826.2023.10423706
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
Sudin, I. A. A., Rahmatullah, B., Abdullah, M. F. W., Tamrin, K. F., Khairudin, M., & Yahya, S. R. (2022). Kajian tinjauan literatur sistematik terhadap pendedahan pelajar universiti kepada percetakan 3D sebagai persediaan ke industri: A systematic literature review study on university students’ exposure to 3D printing as preparation for industry. Journal of ICT in Education, 9(1), 48–60. https://doi.org/10.37134/jictie.vol9.1.5.2022
Sundram, S., & Romli, N. (2023). A pilot study to test the reliability and validity of the Research Instrument. Malaysian Journal of Social Sciences and Humanities, 8(3). https://doi.org/10.47405/mjssh.v8i3.2149
Suhag, A. K., Larik, R. S. A., Tagar, A. A., & Solangi, S. R. (2016). Student academic motivation of secondary schools of Khairpur Mir’s. Academic Research International, 7(1), 100–109.
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Teijlingen, E., & Hundley, V. (2002). The importance of pilot studies. Nursing Standard, 16(40), 33–36. https://doi.org/10.7748/ns2002.06.16.40.33.c3214
Tsai, C.-Y., & Lai, Y.-C. (2022). Design and validation of an augmented reality teaching system for primary logic programming education. Sensors, 22(1), 389. https://doi.org/10.3390/s22010389
Yildiz, F. (2021). Use of euphemisms in youth language. Dil ve Dilbilimi Çalışmaları Dergisi, 17(2), 1117–1128. https://doi.org/10.17263/jlls.904133
Vinnervik, P. (2023). An in-depth analysis of programming in the Swedish school curriculum—rationale, knowledge content and teacher guidance. Journal of Computers in Education, 10(2), 237–271. https://doi.org/10.1007/s40692-022-00230-2
Wen, F.-H., Wu, T., & Hsu, W.-C. (2023). Toward improving student motivation and performance in introductory programming learning by scratch: The role of achievement emotions. Science Progress, 106(4). https://doi.org/10.1177/00368504231205985
Yong, S. T., & Tiong, K. M. (2022). A blended learning approach. International Journal of Information and Communication Technology Education, 18(1), 1–16. https://doi.org/10.4018/ijicte.301276
Kao, G., & Ruan, C.-A. (2022). Designing and evaluating a high interactive augmented reality system for Programming Learning. Computers in Human Behavior, 132, 107245. https://doi.org/10.1016/j.chb.2022.107245
Zainal Abidin, Z., & Abdullah Zawawi, M. A. (2020). Oop-AR: Learn object-oriented programming using augmented reality. International Journal of Multimedia and Recent Innovation, 2(1), 60–75. https://doi.org/10.36079/lamintang.ijmari-0201.83
Zainil, M., Kenedi, A. K., Rahmatina, Indrawati, T., & Handrianto, C. (2022). The influence of a STEM-based digital classroom learning model and high-order thinking skills on the 21st-century skills of elementary school students in Indonesia. Journal of Education and E-Learning Research, 10(1), 29–35. https://doi.org/10.20448/jeelr.v10i1.4336
Zainuddin, Z., Chu, S. K., Shujahat, M., & Perera, C. J. (2020). The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 30, 100326. https://doi.org/10.1016/j.edurev.2020.100326
Zhang, Q., & Yu, Z. (2022). Meta-analysis on investigating and comparing the effects on learning achievement and motivation for gamification and game-based learning. Education Research International, 1–19. https://doi.org/10.1155/2022/1519880
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Salini Krishna Pillai, Noor Hidayah Che Lah, Md Meem Hossain
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
