Development of the Fuzzy Delphi Instrument for Identifying Elements of the PhyARECi-STEM Learning Module
DOI:
https://doi.org/10.37134/ejsmt.vol12.sp.4.2025Keywords:
Augmented Reality, Applied Physics, Fuzzy Delphi Instruments , Learning Module, STEMAbstract
This research aims to describe the process of developing the Fuzzy Delphi instrument to identify elements of the applied physics learning module integrated with STEM and Augmented Reality or AR (PhyARECi-STEM learning module). This questionnaire was developed based on the national standards of higher education analysis in Indonesia, Hilda Taba's curriculum model, module elements found in literature, interviews, and focus groups with experts. The study is descriptive. The 50-item questionnaire relates to learning outcomes, content, process, and assessment. Nine experts conducted the validity evidence of the Fuzzy Delphi questionnaire to ensure content validity. Subsequently, the questionnaire was distributed to 30 respondents to obtain reliability. The data analysis techniques used for validity and reliability were the Content Validity Index (CVI) and Cronbach's alpha. The study's results demonstrated the validity of the Fuzzy Delphi questionnaire, as evidenced by an I-CVI value within the range of 0.89 to 1, an S-CVI value of 0.99, and a Kappa value within the range of 0.89 to 1.00. Furthermore, the reliability of the questionnaire was also established, with a Cronbach alpha value of 0.974. Therefore, the Fuzzy Delphi instrument, validated and demonstrated to be reliable, is now ready to be employed to identify the fundamental elements of the PhyARECi-STEM learning module via expert consensus.
Downloads
References
[1] Amalya, C. P., Artika, W., Safrida, S., Nurmaliah, C., Muhibbuddin, M., & Syukri, M. (2021). Implementation of the problem base learning model combined with E-STEM based student worksheets on learning outcomes and self efficacy on environmental pollution materials. Jurnal Penelitian Pendidikan IPA, 7(1), 37-38. https://doi.org/10.29303/jppipa.v7ispecialissue.962.
[2] Çeliker, H. D. (2020). The Effects of Scenario-Based STEM Project Design Process with Pre-Service Science Teachers: 21st Century Skills and Competencies, Integrative STEM Teaching Intentions and STEM Attitudes. Journal of Educational Issues, 6(2), 451-477. https://doi.org/10.5296/jei.v6i2.17993.
[3] Chen, K., & Chen, C. (2021). Effects of STEM Inquiry Method on Learning Attitude and Creativity. Eurasia Journal of Mathematics, Science and Technology Education, 17(11). https://doi.org/10.29333/EJMSTE/11254.
[4] Oschepkov, A. A., Kidinov, A. V., Babieva, N. S., Vrublevskiy, A. S., Egorova, E. V., & Zhdanov, S. P. (2022). STEM technology-based model helps create an educational environment for developing students' technical and creative thinking. Eurasia Journal of Mathematics, Science and Technology Education, 18(5), em2110. https://doi.org/10.29333/ejmste/12033.
[5] Donmez, I. (2020). STEM education dimensions: From STEM literacy to STEM assessment. Research Highlights in Education and Science 2020, 154. https://files.eric.ed.gov/fulltext/ED611813.pdf#page=158.
[6] Roehrig, G. H., Dare, E. A., Ellis, J. A., & Ring-Whalen, E. (2021). Beyond the basics: A detailed conceptual framework of integrated STEM. Disciplinary and Interdisciplinary Science Education Research, 3, 1-18. https://doi.org/10.1186/s43031-021-00041-y.
[7] Cai, S., Liu, C., Wang, T., Liu, E., & Liang, J. C. (2021). Effects of learning physics using Augmented Reality on students’ self‐efficacy and conceptions of learning. British Journal of Educational Technology, 52(1), 235-251. https://doi.org/10.1111/bjet.13020.
[8] Fidan, M., & Tuncel, M. (2019). Integrating augmented reality into problem based learning: The effects on learning achievement and attitude in physics education. Computers & Education, 142, 103635. https://doi.org/10.1016/j.compedu.2019.103635.
[9] Ropawandi, D., Halim, L., & Husnin, H. (2022). Augmented reality (AR) technology-based learning: the effect on physics learning during the covid-19 pandemic. International Journal of Information and Education Technology, 12(2), 132-140. https://doi.org/10.18178/ijiet.2022.12.2.1596.
[10] Verawati, N., Ernita, N., & Prayogi, S. (2022). Enhancing the reasoning performance of STEM students in modern physics courses using virtual simulation in the LMS platform. International Journal of Emerging Technologies in Learning (iJET), 17(13), 267-277. https://doi.org/10.3991/ijet.v17i13.31459.
[11] Samsudın, M. A., Jamalı, S. M., Zaın, A. N. M., & Ebrahım, N. A. (2020). The effect of STEM project based learning on self-efficacy among high-school physics students. Journal of Turkish Science Education, 17(1), 94-108. https://doi.org/10.2139/ssrn.3574024.
[12] Nugraha, M. G., Kidman, G., & Tan, H. (2023). Pre-service teacher in STEM education: An integrative review and mapping of the Indonesian research literature. Eurasia Journal of Mathematics, Science and Technology Education, 19(5), em2262. https://doi.org/ 10.29333/ejmste/13155.
[13] Aisyah, N., Nawawi, E., & Yusup, M. (2020). How to support pre-service teachers’ conceptual understanding using hyperthermia as STEM context: a case study in sciences education department Universitas Sriwijaya. In Journal of Physics: Conference Series (Vol. 1480, No. 1, p. 012072). IOP Publishing. https://doi.org/10.1088/1742-6596/1480/1/012072.
[14] Widiastuti, I., Budiyanto, C. W., & Ramli, M. (2020). Pre-service teacher conceptions of integrated STEM using the engineering design approach. In 2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 106-111). IEEE. https://doi.org/10.1109/TALE48869.2020.9368382.
[15] Houghton, J. (2023). Learning modules: problem-based learning, blended learning and flipping the classroom. The Law Teacher, 57(3), 271-294. https://doi.org/10.1080/03069400.2023.2208017.
[16] Al Mamun, M. A., Lawrie, G., & Wright, T. (2020). Instructional design of scaffolded online learning modules for self-directed and inquiry-based learning environments. Computers & Education, 144, 103695. https://doi.org/10.1016/j.compedu.2019.103695.
[17] Logan, R. M., Johnson, C. E., & Worsham, J. W. (2021). Development of an e-learning module to facilitate student learning and outcomes. Teaching and Learning in Nursing, 16(2), 139-142. https://doi.org/10.1016/j.teln.2020.10.007.
[18] Malek, N. A., Abdullah, N. S. Y., Mat Darus, M., & Nursulistiyo, E. (2022). A need analysis for the development of Physics game-based interactive module in matriculation college. EDUCATUM Journal of Science, Mathematics and Technology, 9, 48-60. https://doi.org/ 10.37134/ejsmt.vol9.sp.6.2022.
[19] Septiadevana, R., & Abdullah, N. (2024). Developing STEM project-based learning module for primary school teachers: a need analysis. Int J Eval & Res Educ, 13(4), 2585-2593. https://doi.org/10.11591/ijere.v13i4.28894.
[20] Markus, L., & Ishak, M. Z. (2022). Development Of The Secondary School Quantum Physics-Stem (SSQP-Stem) Instructional Module For Physics Teachers: A Fuzzy Delphi Method (FDM) Approach. International Research Journal of Education and Sciences (IRJES), 6 (2), 16–29. http://malrep.uum.edu.my/rep/Record/my.ums.eprints.38260.
[21] Al-Rikabi, Y. K., & Montazer, G. A. (2024). Designing an e-learning readiness assessment model for Iraqi universities employing Fuzzy Delphi Method. Education and Information Technologies, 29(2), 2217-2257. https://doi.org/10.1007/s10639-023-11889-0.
[22] Karayağiz Muslu, G., Manav, G., Carti Dörterler, Ö., & Kara, R. (2024). Investigation of the psychometric properties of the Turkish child-friendly Oral Health Questionnaire. BMC Oral Health, 24(1), 995. https://doi.org/10.1186/s12903-024-04764-y.
[23] Kudsi, Z., Fenlon, M. R., Hill, K., & Baysan, A. (2020). Developing a questionnaire to measure psychological disturbance associated with tooth loss. Journal of Dentistry, 98, 103353. https://doi.org/10.1016/j.jdent.2020.103353.
[24] Ibrahim, M. Y. (2025). The most appropriate scale for behavioral research is a seven-point rating. Multidisciplinary Reviews, 8(4), 2025126-2025126. https://doi.org/10.31893/multirev.2025126.
[25] Noh, N. M., Siraj, S., Halili, S. H., Jamil, M. R. M., & Husin, Z. (2019). Application of fuzzy delphi method as a vital element in technology as a tool in design thinking based learning. Asia Pacific Journal of Educators and Education, 34, 129-151. https://doi.org/10.21315/apjee2019.34.7.
[26] López-Belmonte, J., Pozo-Sánchez, S., Lampropoulos, G., & Moreno-Guerrero, A. J. (2022). Design and validation of a questionnaire for the evaluation of educational experiences in the metaverse in Spanish students (METAEDU). Heliyon, 8(11). https://doi.org/10.1016/j.heliyon.2022.e11364.
[27] Silva, W. A. D., de Sampaio Brito, T. R., & Pereira, C. R. (2022). COVID-19 anxiety scale (CAS): Development and psychometric properties. Current Psychology, 41(8), 5693-5702. https://doi.org/10.1007/s12144-020-01195-0.
[28] Elangovan, N., & Sundaravel, E. (2021). Method of preparing a document for survey instrument validation by experts. MethodsX, 8, 101326. https://doi.org/10.1016/j.mex.2021.101326.
[29] Cocks, K., Wells, J. R., Johnson, C., Schmidt, H., Koller, M., Oerlemans, S., ... & European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group. (2023). Content validity of the EORTC quality of life questionnaire QLQ-C30 for use in cancer. European journal of cancer, 178, 128-138. https://doi.org/10.1016/j.ejca.2022.10.026.
[30] Guillot-Valdés, M., Guillén-Riquelme, A., & Buela-Casal, G. (2022). Content validity through expert judgment for the Depression Clinical Evaluation Test. International Journal of Clinical and Health Psychology, 22(2), 100292. https://doi.org/10.1016/j.ijchp.2022.100292.
[31] Roebianto, A., Savitri, S. I., Aulia, I., Suciyana, A., & Mubarokah, L. (2023). Content validity: Definition and procedure of content validation in psychological research. TPM, 30(1), 5-18. https://doi.org/10.4473/TPM30.1.1.
[32] Ramly, S. N. F., Ahmad, N. J., & Yakob, N. (2022). Development, validity, and reliability of chemistry scientific creativity test for pre-university students. International Journal of Science Education, 44(14), 1-16. https://doi.org/ 10.1080/09500693.2022.2116298.
[33] Yusoff, M. S. B. (2019). ABC of content validation and content validity index calculation. Education in medicine journal, 11(2), 49-54. https://doi.org/10.21315/eimj2019.11.2.6.
[34] Farkas, B. C., Krajcsi, A., Janacsek, K., & Nemeth, D. (2024). The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task. Behavior Research Methods, 56(1), 301-317. https://doi.org/10.3758/s13428-022-02038-5.
[35] Vosloo, C., & Van Staden, W. (2024). Validity and reliability of the agentive steadfastness index. International Journal of Psychology, 59(3), 378-389. https://doi.org/10.1002/ijop.13098.
[36] Zadorozhny, B. S., Petrides, K. V., Jongerling, J., Cuppello, S., & van der Linden, D. (2024). Assessing the temporal stability of a measure of trait emotional intelligence: Systematic review and empirical analysis. Personality and Individual Differences, 217, 112467. https://doi.org/10.1016/j.paid.2023.112467.
[37] Handage, S., & Chander, M. (2021). Development of an instrument for measuring the student learning outcomes: A content validation process. Indian Journal of Extension Education, 57(3), 1-7. https://doi.org/10.48165/ijee.2021.57302.
[38] Dalawi, I., Isa, M. R., Chen, X. W., Azhar, Z. I., & Aimran, N. (2023). Development of the Malay Language of understanding, attitude, practice and health literacy questionnaire on COVID-19 (MUAPHQ C-19): content validity & face validity analysis. BMC Public Health, 23(1), 1131. https://doi.org/10.1186/s12889-023-16044-5.
[39] Gizaw, Z., Yalew, A. W., Bitew, B. D., Lee, J., & Bisesi, M. (2022). Development and validation of questionnaire to assess exposure of children to enteric infections in the rural northwest Ethiopia. Scientific Reports, 12(1), 6740. https://doi.org/10.1038/s41598-022-10811-x.
[40] Roy, R., Sukumar, G. M., Philip, M., & Gopalakrishna, G. (2023). Face, content, criterion and construct validity assessment of a newly developed tool to assess and classify work–related stress (TAWS–16). Plos one, 18(1), e0280189. https://doi.org/10.1371/journal.pone.0280189.
[41] Özkaya, G., Aydin, M. O., & Alper, Z. (2021). Distance education perception scale for medical students: a validity and reliability study. BMC medical education, 21, 1-8. https://doi.org/10.1186/s12909-021-02839-w.
[42] Akbari-Chehrehbargh, Z., Sadat Tavafian, S., & Montazeri, A. (2020). The Back-care Behavior Assessment Questionnaire (BABAQ) for schoolchildren: development and psychometric evaluation. BMC Public Health, 20, 1-10. https://doi.org/10.1186/s12889-020-09318-9.
[43] Kennedy, I. (2022). Sample size determination in test-retest and Cronbach alpha reliability estimates. British Journal of Contemporary Education, 2(1), 17-29. https://doi.org/10.52589/bjce-fy266hk9.
[44] Kumar, A., Rajendran, A., Usman, M., Ahuja, J., Samad, S., Mittal, A., ... & Wig, N. (2022). Development and validation of a questionnaire to evaluate the knowledge, attitude and practices regarding travel medicine amongst physicians in an apex tertiary hospital in Northern India. Tropical Diseases, Travel Medicine and Vaccines, 8(1), 13. https://doi.org/10.1186/s40794-022-00170-w.
[45] Govindasamy, P., Cumming, T. M., & Abdullah, N. (2024). Validity and reliability of a needs analysis questionnaire for the development of a creativity module. Journal of Research in Special Educational Needs. https://doi.org/10.1111/1471-3802.12659.
[46] Masuwai, A., Zulkifli, H., & Hamzah, M. I. (2024). Evaluation of content validity and face validity of secondary school Islamic education teacher self-assessment instrument. Cogent Education, 11(1), 2308410. https://doi.org/10.1080/2331186X.2024.2308410
[47] Shrotryia, V. K., & Dhanda, U. (2019). Content validity of assessment instrument for employee engagement. Sage Open, 9(1). https://doi.org/10.1177/2158244018821751.
[48] Zulkifli, H., Ismail, A. M., Muhamad, N. A. F., Hussin, N. H., & Yusuf, S. A. M. (2024). Reliability and Validity of The Fuzzy Delphi Instrument of Islamic Cognitive Domains Running Head: Reliability and Validity of Islamic Cognitive Domains. International Journal of Academic Research in Progressive Education and Development, 13(2). https://doi.org/10.6007/ijarped/v13-i2/21201.
[49] Taherdoost, H. (2016). Validity and reliability of the research instrument; how to test the validation of a questionnaire/survey in a research. International Journal of Academic Research in Management (IJARM), 5. https://doi.org/10.2139/ssrn.3205040.
[50] Von Soest, C. (2023). Why do we speak to experts? Reviving the strength of the expert interview method. Perspectives on Politics, 21(1), 277-287. https://doi.org/10.1017/S1537592722001116.
[51] Sand, P., Kleiberg, A. N., Kljajić, M., & Lannering, B. (2020). The reliability of the health related quality of life questionnaire PedsQL 3.0 cancer module in a sample of Swedish children. BMC pediatrics, 20, 1-8. https://doi.org/10.1186/s12887-020-02387-0.
[52] Zhang, J. Y., Wang, Y., Wan, B. J., Qi, L. H., Shu, T., & Feng, Z. C. (2023). Clinicians’ response ability for public health emergencies: a questionnaire development and validation study. Journal of Public Health, 1-7. https://doi.org/10.1007/s10389-023-01951-z.
[53] Schmid, M., Brianza, E., & Petko, D. (2020). Developing a short assessment instrument for Technological Pedagogical Content Knowledge (TPACK. xs) and comparing the factor structure of an integrative and a transformative model. Computers & Education, 157, 103967. https://doi.org/10.1016/j.compedu.2020.103967.
[54] Castillejos-Carrasco-Muñoz, R., Peinado-Rubia, A. B., Lérida-Ortega, M. Á., Ibáñez-Vera, A. J., Tapia-Toca, M. C., & Lomas-Vega, R. (2023). Validity and reliability of the Niigata PPPD Questionnaire in a Western population. European Archives of Oto-Rhino-Laryngology, 280(12), 5267-5276. https://doi.org/10.1007/s00405-023-08038-1.
[55] Sims, J., Sloesen, B., Bentley, S., Naujoks, C., Arbuckle, R., Chiva-Razavi, S., ... & Wolffsohn, J. S. (2024). Psychometric evaluation of the near activity visual questionnaire presbyopia (NAVQ-P) and additional patient-reported outcome items. Journal of Patient-Reported Outcomes, 8(1), 1-17. https://doi.org/10.1186/s41687-024-00717-9.
[56] Bom, J. A., Voormolen, D. C., Brouwer, W. B., de Bekker-Grob, E. W., & van Exel, J. (2024). Construct Validity, Reliability, and Responsiveness of the 10-Item Well-being Instrument for Use in Economic Evaluation Studies. Value in Health. https://doi.org/10.1016/j.jval.2024.02.014.
[57] Louis, E., Lee, W. J., Litcher-Kelly, L., Ollis, S., Pranschke, E., Fitzgerald, K., ... & Panés, J. (2024). Content Validity and Psychometric Evaluation of the Crohn’s Symptom Severity (CSS) Questionnaire in Patients with Moderately to Severely Active Crohn’s Disease. Advances in Therapy, 41(9), 3678-3705. https://doi.org/10.1007/s12325-024-02923-9.
[58] Fokkens, W., Trigg, A., Lee, S. E., Chan, R. H., Diamant, Z., Hopkins, C., ... & Tabberer, M. (2023). Mepolizumab improvements in health-related quality of life and disease symptoms in a patient population with very severe chronic rhinosinusitis with nasal polyps: psychometric and efficacy analyses from the SYNAPSE study. Journal of patient-reported outcomes, 7(1), 4. https://doi.org/10.1186/s41687-023-00543-5.
[59] López, M. (2023). The effect of sampling mode on response rate and bias in elite surveys,” Qual. Quant., 57 (2), 1303–1319. https://doi.org/10.1007/s11135-022-01406-9.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Renol Afrizon, Lilia Ellany Mohtar, Mohd Syahriman Mohd Azmi
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
