Date Log
Copyright (c) 2024 Muhammad Galuh Elga Romadhoan
This work is licensed under a Creative Commons Attribution 4.0 International License.
Has AI cracked the code on English proficiency assessments? A look at how AI is revolutionizing writing evaluations
Corresponding Author(s) : Muhammad Galuh Elga Romadhoan
Journal of Qualitative Research in Language Education,
Vol. 1 No. 1 (2024): June 2024
Abstract
This scholarly inquiry fuses the rapid growth in artificially intelligent writing systems with associated challenges in assessment tools that gauge English compositional skills. To put it another way, in some contexts, many AI-written texts often exhibit human quality. However, inconsistencies have been found which can problematize identification in differentiating human- from machine-penned compositions. It follows through with the claim that the availability of AI writing instruments implies score potential inflation for any type of standardized English language proficiency examination, which in turn challenges its validity. With this view in mind, 5-7 language assessment experts have been approached in this qualitative study for interviews and document analyses to elucidate concerns connected with differentiating human writing from AI writing. Submissions from cutting-edge AI models are matched against known criteria to identify singular features related to coherence, style, and originality, including the handling of open-ended tasks. Thematic analysis was tease out key themes related to the variation in the quality of the composition. Results offer useful insights into possible future detection methods and assessment designs.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- Abida, F. I. N., Kuswardani, R., Purwati, O., Rosyid, A., & Minarti, E. (2023). Assessing Language Proficiency through AI Chatbot-Based Evaluations. Proceedings of International Conference on Islamic Civilization and Humanities, 1, 138–145.
- Aggarwal, D. (2023). Integration of innovative technological developments and AI with education for an adaptive learning pedagogy. China Petroleum Processing and Petrochemical Technology, 23(2), 709–714.
- Alieto, E., Abequibel-Encarnacion, B., Estigoy, E., Balasa, K., Eijansantos, A., & Torres-Toukoumidis, A. (2024). Teaching inside a digital classroom: A quantitative analysis of attitude, technological competence and access among teachers across subject disciplines. Heliyon, 10(2), e24282. https://doi.org/10.1016/j.heliyon.2024.e24282
- Amirjalili, F., Neysani, M., & Nikbakht, A. (2024). Exploring the boundaries of authorship: a comparative analysis of AI-generated text and human academic writing in English literature. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1347421
- Creely, E. (2024). Exploring the Role of Generative AI in Enhancing Language Learning: Opportunities and Challenges. International Journal of Changes in Education. https://doi.org/10.47852/bonviewIJCE42022495
- Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023). From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), 615–622. https://doi.org/10.5114/biolsport.2023.125623
- Dong, Y. (2023). Revolutionizing Academic English Writing through AI-Powered Pedagogy: Practical Exploration of Teaching Process and Assessment. Journal of Higher Education Research, 4(2), 52. https://doi.org/10.32629/jher.v4i2.1188
- Dr. Neeraj Yadav. (2024). The Impact of Digital Learning on Education. International Journal of Multidisciplinary Research in Arts, Science and Technology, 2(1), 24–34. https://doi.org/10.61778/ijmrast.v2i1.34
- Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research, 25(3), 277–304. https://doi.org/10.1080/15228053.2023.2233814
- Hadi Mogavi, R., Deng, C., Juho Kim, J., Zhou, P., D. Kwon, Y., Hosny Saleh Metwally, A., Tlili, A., Bassanelli, S., Bucchiarone, A., Gujar, S., Nacke, L. E., & Hui, P. (2024). ChatGPT in education: A blessing or a curse? A qualitative study exploring early adopters’ utilization and perceptions. Computers in Human Behavior: Artificial Humans, 2(1), 100027. https://doi.org/10.1016/j.chbah.2023.100027
- Han, B., Nawaz, S., Buchanan, G., & McKay, D. (2023). Ethical and Pedagogical Impacts of AI in Education (pp. 667–673). https://doi.org/10.1007/978-3-031-36272-9_54
- Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022). Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education. Mathematical Problems in Engineering, 2022, 1–19. https://doi.org/10.1155/2022/5215722
- Hz, B. I. R., Laiya, R. E., & Sarumaha, M. S. (2023). Navigating English Writing Proficiency Tests in the Era of Artificial Intelligence. Journal of English Education and Teaching, 7(3), 480–498.
- Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
- Marquez, R., Barrios, N., Vera, R. E., Mendez, M. E., Tolosa, L., Zambrano, F., & Li, Y. (2023). A perspective on the synergistic potential of artificial intelligence and product-based learning strategies in biobased materials education. Education for Chemical Engineers, 44, 164–180. https://doi.org/10.1016/j.ece.2023.05.005
- Mennella, C., Maniscalco, U., De Pietro, G., & Esposito, M. (2024). Ethical and regulatory challenges of AI technologies in healthcare: A narrative review. Heliyon, 10(4), e26297. https://doi.org/10.1016/j.heliyon.2024.e26297
- Mohamed, Y. A., Khanan, A., Bashir, M., Mohamed, A. H. H. M., Adiel, M. A. E., & Elsadig, M. A. (2024). The Impact of Artificial Intelligence on Language Translation: A Review. IEEE Access, 12, 25553–25579. https://doi.org/10.1109/ACCESS.2024.3366802
- Moorhouse, B. L., Yeo, M. A., & Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world’s top-ranking universities. Computers and Education Open, 5, 100151. https://doi.org/10.1016/j.caeo.2023.100151
- Moroianu, N., Iacob, S.-E., & Constantin, A. (2023). Artificial Intelligence in Education: a Systematic Review. In Geopolitical perspectives and technological challenges for sustainable growth in the 21st century (pp. 906–921). Sciendo. https://doi.org/10.2478/9788367405546-084
- Park, Y., & Doo, M. Y. (2024). Role of AI in Blended Learning: A Systematic Literature Review. International Review of Research in Open and Distributed Learning, 25(1), 164–196.
- Pratama, R. M. D., & Hastuti, D. P. (2024). The use of artificial intelligence to improve EFL students’ writing skill. English Learning Innovation, 5(1), 13–25.
- Raheem, B. R., Anjum, F., & Ghafar, Z. N. (2023). Exploring the profound impact of artificial intelligence applications (Quillbot, Grammarly and ChatGPT) on English academic writing: A Systematic Review. International Journal of Integrative Research (IJIR), 1(10), 599–622.
- Rahmi, R., Amalina, Z., Andriansyah, A., & Rodgers, A. (2024). Does it really help? Exploring the impact of Al-Generated writing assistant on the students’ English writing. Studies in English Language and Education, 11(2), 998–1012. https://doi.org/10.24815/siele.v11i2.35875
- Reviewing the performance of AI detection tools in differentiating between AI-generated and human-written texts: A literature and integrative hybrid review. (2024). Journal of Applied Learning & Teaching, 7(1). https://doi.org/10.37074/jalt.2024.7.1.14
- Rezai, A., Namaziandost, E., Miri, M., & Kumar, T. (2022). Demographic biases and assessment fairness in classroom: insights from Iranian university teachers. Language Testing in Asia, 12(1), 8. https://doi.org/10.1186/s40468-022-00157-6
- Rizqa, K., Drajati, N. A., & Putra, K. A. (2022). PRE-SERVICE TEACHERS’PERCEPTIONS ON PEER ASSESSMENT PROCESS IN LEARNING ACADEMIC WRITING: ARTIFICIAL INTELLIGENCE (AI). ICONELT, 3, 50–58.
- Roy, S., & Paul, S. K. (2023). Revolutionizing Education: How Artificial Intelligence is transforming the Learning Landscape. Int. J. Trend Sci. Res. Dev, 7, 748–756.
- Shanorra, T. T., Sofyan, R., & Sumbayak, D. M. (2021). A writing skill assessment of the first semester English department students of the Universitas Sumatera Utara. VISION, 17(2).
- Smagorinsky, P., Daigle, E. A., Allen, C. O., & Bynum, S. (2010). Bullshit in academic writing: A protocol analysis of a high school senior’s process of interpreting Much Ado About Nothing. Research in the Teaching of English, 44(4), 368–405.
- Toba, R., Noor, W. N., & Sanu, L. O. (2019). The Current Issues of Indonesian EFL Students’ Writing Skills: Ability, Problem, and Reason in Writing Comparison and Contrast Essay. Dinamika Ilmu, 57–73. https://doi.org/10.21093/di.v19i1.1506
- Toncic, J. (2020). Teachers, AI grammar checkers, and the newest literacies: Emending writing pedagogy and assessment. Digital Culture & Education, 12(1), 26.
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z
- Wulandari, F., Astuti, M. T., & Marhamah, M. (2024). Enhancing writing literacy teachers’ through AI development. Jurnal Onoma: Pendidikan, Bahasa Dan Sastra, 10(1), 246–256.
- Zhao, R., Zhuang, Y., Zou, D., Xie, Q., & Yu, P. L. H. (2023). AI-assisted automated scoring of picture-cued writing tasks for language assessment. Education and Information Technologies, 28(6), 7031–7063. https://doi.org/10.1007/s10639-022-11473-y
References
Abida, F. I. N., Kuswardani, R., Purwati, O., Rosyid, A., & Minarti, E. (2023). Assessing Language Proficiency through AI Chatbot-Based Evaluations. Proceedings of International Conference on Islamic Civilization and Humanities, 1, 138–145.
Aggarwal, D. (2023). Integration of innovative technological developments and AI with education for an adaptive learning pedagogy. China Petroleum Processing and Petrochemical Technology, 23(2), 709–714.
Alieto, E., Abequibel-Encarnacion, B., Estigoy, E., Balasa, K., Eijansantos, A., & Torres-Toukoumidis, A. (2024). Teaching inside a digital classroom: A quantitative analysis of attitude, technological competence and access among teachers across subject disciplines. Heliyon, 10(2), e24282. https://doi.org/10.1016/j.heliyon.2024.e24282
Amirjalili, F., Neysani, M., & Nikbakht, A. (2024). Exploring the boundaries of authorship: a comparative analysis of AI-generated text and human academic writing in English literature. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1347421
Creely, E. (2024). Exploring the Role of Generative AI in Enhancing Language Learning: Opportunities and Challenges. International Journal of Changes in Education. https://doi.org/10.47852/bonviewIJCE42022495
Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023). From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), 615–622. https://doi.org/10.5114/biolsport.2023.125623
Dong, Y. (2023). Revolutionizing Academic English Writing through AI-Powered Pedagogy: Practical Exploration of Teaching Process and Assessment. Journal of Higher Education Research, 4(2), 52. https://doi.org/10.32629/jher.v4i2.1188
Dr. Neeraj Yadav. (2024). The Impact of Digital Learning on Education. International Journal of Multidisciplinary Research in Arts, Science and Technology, 2(1), 24–34. https://doi.org/10.61778/ijmrast.v2i1.34
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research, 25(3), 277–304. https://doi.org/10.1080/15228053.2023.2233814
Hadi Mogavi, R., Deng, C., Juho Kim, J., Zhou, P., D. Kwon, Y., Hosny Saleh Metwally, A., Tlili, A., Bassanelli, S., Bucchiarone, A., Gujar, S., Nacke, L. E., & Hui, P. (2024). ChatGPT in education: A blessing or a curse? A qualitative study exploring early adopters’ utilization and perceptions. Computers in Human Behavior: Artificial Humans, 2(1), 100027. https://doi.org/10.1016/j.chbah.2023.100027
Han, B., Nawaz, S., Buchanan, G., & McKay, D. (2023). Ethical and Pedagogical Impacts of AI in Education (pp. 667–673). https://doi.org/10.1007/978-3-031-36272-9_54
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022). Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education. Mathematical Problems in Engineering, 2022, 1–19. https://doi.org/10.1155/2022/5215722
Hz, B. I. R., Laiya, R. E., & Sarumaha, M. S. (2023). Navigating English Writing Proficiency Tests in the Era of Artificial Intelligence. Journal of English Education and Teaching, 7(3), 480–498.
Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
Marquez, R., Barrios, N., Vera, R. E., Mendez, M. E., Tolosa, L., Zambrano, F., & Li, Y. (2023). A perspective on the synergistic potential of artificial intelligence and product-based learning strategies in biobased materials education. Education for Chemical Engineers, 44, 164–180. https://doi.org/10.1016/j.ece.2023.05.005
Mennella, C., Maniscalco, U., De Pietro, G., & Esposito, M. (2024). Ethical and regulatory challenges of AI technologies in healthcare: A narrative review. Heliyon, 10(4), e26297. https://doi.org/10.1016/j.heliyon.2024.e26297
Mohamed, Y. A., Khanan, A., Bashir, M., Mohamed, A. H. H. M., Adiel, M. A. E., & Elsadig, M. A. (2024). The Impact of Artificial Intelligence on Language Translation: A Review. IEEE Access, 12, 25553–25579. https://doi.org/10.1109/ACCESS.2024.3366802
Moorhouse, B. L., Yeo, M. A., & Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world’s top-ranking universities. Computers and Education Open, 5, 100151. https://doi.org/10.1016/j.caeo.2023.100151
Moroianu, N., Iacob, S.-E., & Constantin, A. (2023). Artificial Intelligence in Education: a Systematic Review. In Geopolitical perspectives and technological challenges for sustainable growth in the 21st century (pp. 906–921). Sciendo. https://doi.org/10.2478/9788367405546-084
Park, Y., & Doo, M. Y. (2024). Role of AI in Blended Learning: A Systematic Literature Review. International Review of Research in Open and Distributed Learning, 25(1), 164–196.
Pratama, R. M. D., & Hastuti, D. P. (2024). The use of artificial intelligence to improve EFL students’ writing skill. English Learning Innovation, 5(1), 13–25.
Raheem, B. R., Anjum, F., & Ghafar, Z. N. (2023). Exploring the profound impact of artificial intelligence applications (Quillbot, Grammarly and ChatGPT) on English academic writing: A Systematic Review. International Journal of Integrative Research (IJIR), 1(10), 599–622.
Rahmi, R., Amalina, Z., Andriansyah, A., & Rodgers, A. (2024). Does it really help? Exploring the impact of Al-Generated writing assistant on the students’ English writing. Studies in English Language and Education, 11(2), 998–1012. https://doi.org/10.24815/siele.v11i2.35875
Reviewing the performance of AI detection tools in differentiating between AI-generated and human-written texts: A literature and integrative hybrid review. (2024). Journal of Applied Learning & Teaching, 7(1). https://doi.org/10.37074/jalt.2024.7.1.14
Rezai, A., Namaziandost, E., Miri, M., & Kumar, T. (2022). Demographic biases and assessment fairness in classroom: insights from Iranian university teachers. Language Testing in Asia, 12(1), 8. https://doi.org/10.1186/s40468-022-00157-6
Rizqa, K., Drajati, N. A., & Putra, K. A. (2022). PRE-SERVICE TEACHERS’PERCEPTIONS ON PEER ASSESSMENT PROCESS IN LEARNING ACADEMIC WRITING: ARTIFICIAL INTELLIGENCE (AI). ICONELT, 3, 50–58.
Roy, S., & Paul, S. K. (2023). Revolutionizing Education: How Artificial Intelligence is transforming the Learning Landscape. Int. J. Trend Sci. Res. Dev, 7, 748–756.
Shanorra, T. T., Sofyan, R., & Sumbayak, D. M. (2021). A writing skill assessment of the first semester English department students of the Universitas Sumatera Utara. VISION, 17(2).
Smagorinsky, P., Daigle, E. A., Allen, C. O., & Bynum, S. (2010). Bullshit in academic writing: A protocol analysis of a high school senior’s process of interpreting Much Ado About Nothing. Research in the Teaching of English, 44(4), 368–405.
Toba, R., Noor, W. N., & Sanu, L. O. (2019). The Current Issues of Indonesian EFL Students’ Writing Skills: Ability, Problem, and Reason in Writing Comparison and Contrast Essay. Dinamika Ilmu, 57–73. https://doi.org/10.21093/di.v19i1.1506
Toncic, J. (2020). Teachers, AI grammar checkers, and the newest literacies: Emending writing pedagogy and assessment. Digital Culture & Education, 12(1), 26.
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z
Wulandari, F., Astuti, M. T., & Marhamah, M. (2024). Enhancing writing literacy teachers’ through AI development. Jurnal Onoma: Pendidikan, Bahasa Dan Sastra, 10(1), 246–256.
Zhao, R., Zhuang, Y., Zou, D., Xie, Q., & Yu, P. L. H. (2023). AI-assisted automated scoring of picture-cued writing tasks for language assessment. Education and Information Technologies, 28(6), 7031–7063. https://doi.org/10.1007/s10639-022-11473-y