Evaluating the Usability of an RF–LSTM Learning Platform with Explainable AI among University Students
DOI:
https://doi.org/10.62238/chatra.v4i3.491Keywords:
explainable AI, learning platform, RF–LSTM, usabilityAbstract
Educational AI platforms require usable interfaces that enable students to interpret complex predictive and explainability outputs. This study evaluated the usability of an RF-LSTM learning platform incorporating explainable artificial intelligence among university students. A multidimensional usability evaluation involved 40 Informatics Engineering and Computer Science students who completed six structured tasks covering data exploration, preprocessing, model prediction, performance comparison, SHAP interpretation, and navigation. Usability evidence was obtained from task performance, structured observations, the System Usability Scale (SUS), and open-ended feedback. Task completion ranged from 85% to 100%, while the mean SUS score of 76.7 indicated positive overall usability. Greater difficulties emerged when students compared performance metrics and interpreted SHAP visualizations, particularly regarding technical terminology, visual presentation, navigation, and system feedback. The findings suggest that algorithmic explainability does not automatically ensure comprehensibility during student interaction. Contextual explanations, clearer navigation cues, scalable SHAP visualizations, and processing feedback emerged as design priorities, although the findings remain limited to students within a higher education context.
Platform pendidikan berbasis AI memerlukan antarmuka yang mudah digunakan agar mahasiswa mampu menginterpretasikan keluaran prediktif dan explainability yang kompleks. Penelitian ini mengevaluasi usabilitas platform pembelajaran RF-LSTM yang mengintegrasikan explainable artificial intelligence pada mahasiswa. Evaluasi usabilitas multidimensional melibatkan 40 mahasiswa Teknik Informatika dan Ilmu Komputer yang menyelesaikan enam tugas terstruktur meliputi eksplorasi data, preprocessing, prediksi model, perbandingan performa, interpretasi SHAP, dan navigasi. Bukti usabilitas diperoleh melalui performa tugas, observasi terstruktur, System Usability Scale (SUS), dan umpan balik terbuka. Tingkat penyelesaian tugas berkisar antara 85% hingga 100%, sedangkan rerata skor SUS sebesar 76,7 menunjukkan usabilitas keseluruhan yang positif. Kesulitan lebih terlihat ketika mahasiswa membandingkan metrik performa dan menginterpretasikan visualisasi SHAP, terutama terkait terminologi teknis, penyajian visual, navigasi, dan umpan balik sistem. Temuan menunjukkan bahwa explainability algoritmik tidak secara otomatis menjamin keterpahaman dalam interaksi mahasiswa. Penjelasan kontekstual, penanda navigasi yang lebih jelas, visualisasi SHAP yang dapat disesuaikan, dan umpan balik pemrosesan menjadi prioritas pengembangan, meskipun temuan terbatas pada mahasiswa dalam konteks perguruan tinggi.
References
Afzaal, M., Zia, A., Nouri, J., & Fors, U. (2024). Informative Feedback and Explainable AI-Based Recommendations to Support Students’ Self-regulation. Technology, Knowledge and Learning, 29, 331–354. https://doi.org/10.1007/s10758-023-09650-0
Alfredo, R., Echeverria, V., Jin, Y., Yan, L., Swiecki, Z., Gašević, D., & Martinez-Maldonado, R. (2024). Human-centred learning analytics and AI in education: A systematic literature review. Computers and Education: Artificial Intelligence, 6, 100215. https://doi.org/10.1016/j.caeai.2024.100215
Bobek, S., Korycińska, P., Krakowska, M., Mozolewski, M., Rak, D., Zych, M., Wójcik, M., & Nalepa, G. J. (2025). User-centric evaluation of explainability of AI with and for humans: a comprehensive empirical study. International Journal of Human-Computer Studies, 103625.
Bodily, R., & Verbert, K. (2017a). Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172
Bodily, R., & Verbert, K. (2017b). Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133.
Byrne, D. (2022). A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Quality & Quantity, 56, 1391–1412. https://doi.org/10.1007/s11135-021-01182-y
Castilla, D., Jaen, I., Suso-Ribera, C., Garcia-Soriano, G., Zaragoza, I., Breton-Lopez, J., Mira, A., Diaz-Garcia, A., & Garcia-Palacios, A. (2024). Psychometric properties of the Spanish full and short forms of the system usability scale (SUS): detecting the effect of negatively worded items. International Journal of Human–Computer Interaction, 40(15), 4145–4151.
Clow, D. (2012). The learning analytics cycle. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, 134–138. https://doi.org/10.1145/2330601.2330636
De Barba, P., Oliveira, E. A., & Hu, X. (2022). Same graph, different data: A usability study of a student-facing dashboard based on self-regulated learning theory. ASCILITE 2022: Reconnecting Relationships through Technology. https://doi.org/10.14742/apubs.2022.168
de Vreugd, L., van Leeuwen, A., Jansen, R., & van der Schaaf, M. (2024). Learning Analytics Dashboard Design and Evaluation to Support Student Self-Regulation of Study Behaviour. Journal of Learning Analytics, 11(3), 249–262. https://doi.org/10.18608/jla.2024.8529
De Vreugd, L., Van Leeuwen, A., Jansen, R., & Van der Schaaf, M. (2024). Learning Analytics Dashboard Design and Evaluation to Support Student Self-Regulation of Study Behaviour. Journal of Learning Analytics, 1–14. https://doi.org/10.18608/jla.2024.8529
El Marzki, M. (2025). Predicting and Interpreting Student Academic Performance: A Deep Learning and Shapley Additive Explanations Approach. SHS Web of Conferences. https://www.shs-conferences.org/articles/shsconf/abs/2025/05/shsconf_cifem2024_01001/shsconf_cifem2024_01001.html
Georgsson, M., & Staggers, N. (2016). Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. Journal of the American Medical Informatics Association, 23(1), 5–11. https://doi.org/10.1093/jamia/ocv099
Ghimire, S., Abdulla, S., Joseph, L. P., Prasad, S., Murphy, A., Devi, A., Barua, P. D., Deo, R. C., Acharya, R., & Yaseen, Z. M. (2024). Explainable artificial intelligence-machine learning models to estimate overall scores in tertiary preparatory general science course. Computers and Education: Artificial Intelligence, 7, 100331. https://doi.org/10.1016/j.caeai.2024.100331
Guan, Y., Wang, F., & Song, S. (2026). Interpretable machine learning for academic performance prediction: A SHAP-based analysis of key influencing factors. Innovations in Education and Teaching International, 63(5), 1553–1572. https://doi.org/10.1080/14703297.2025.2532050
Hooshyar, D., & Yang, Y. (2024). Problems With SHAP and LIME in Interpretable AI for Education: A Comparative Study of Post-Hoc Explanations and Neural-Symbolic Rule Extraction. IEEE Access, 12, 137472–137490. https://doi.org/10.1109/ACCESS.2024.3463948
Hussain, A., Mkpojiogu, E. O. C., & Hussain, Z. (2015). USABILITY EVALUATION OF A WEB-BASED HEALTH AWARENESS PORTAL ON SMARTPHONE DEVICES USING ISO 9241- 11 MODEL. Jurnal Teknologi, 77(4). https://doi.org/10.11113/jt.v77.6035
Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020a). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. The Internet and Higher Education, 47, 100758. https://doi.org/10.1016/j.iheduc.2020.100758
Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020b). From Students with Love: An Empirical Study on Learner Goals, Self-Regulated Learning and Sense-Making of Learning Analytics in Higher Education. The Internet and Higher Education, 47, 100758. https://doi.org/10.1016/j.iheduc.2020.100758
Kaveri, A., Silvola, A., & Muukkonen, H. (2023). Supporting Student Agency with a Student-Facing Learning Analytics Dashboard: Perceptions of an Interdisciplinary Development Team. Journal of Learning Analytics, 10(2), 85–99. https://doi.org/10.18608/jla.2023.7729
Khosravi, H., Buckingham Shum, S., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martínez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable Artificial Intelligence in education. Computers and Education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable Artificial Intelligence in education. Computers and Education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
Krabbel, H., Munim, Z. H., & Bustgaard, M. (2025). Factors Affecting the System Usability of a Maritime Learning Analytics Dashboard Prototype. In Integrating Emerging Technologies into Education and Training (pp. 11–22). https://doi.org/10.1007/978-3-031-84170-5_2
Lewis, J. R. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction, 34(7), 577–590. https://doi.org/10.1080/10447318.2018.1455307
Liu, L., Zhou, T., Li, C., Huang, Y., Wu, Y., & Chen, D. (2024). The mediating effect of newly graduated nurses’ coping styles between personality traits and transitional shock. Chinese Journal of Nursing, 59(20), 2514–2521. https://doi.org/10.3761/j.issn.0254-1769.2024.20.012
Liu, R., Zhang, L., Cao, Z., & Mi, J. (2024). Influence of gamification affordance on young citizens’ motivation and learning performance toward digital civic education. Journal of Chinese Governance. https://doi.org/10.1080/23812346.2024.2327735
Long, D., & Magerko, B. (2020). What is AI Literacy? Competencies and Design Considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727
Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30 (NIPS 2017), 4765–4774. https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions
Maarif, M. R., Saleh, A. R., Habibi, M., Fitriyani, N. L., & Syafrudin, M. (2023). Energy Usage Forecasting Model Based on Long Short-Term Memory (LSTM) and eXplainable Artificial Intelligence (XAI). Information, 14(5), 265. https://doi.org/10.3390/info14050265
Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2020). A Systematic Review of Empirical Studies on Learning Analytics Dashboards: A Self-Regulated Learning Perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245.
Mubarak, H., Hammoudeh, A., Ahmad, S., Abdellatif, A., Mekhilef, S., Mokhlis, H., & Dupont, S. (2023). A hybrid machine learning method with explicit time encoding for improved Malaysian photovoltaic power prediction. Journal of Cleaner Production, 382, 134979. https://doi.org/10.1016/j.jclepro.2022.134979
Park, Y., & Jo, I.-H. (2019a). Development of the Learning Analytics Dashboard to Support Students’ Learning Performance. Journal of Universal Computer Science, 21(1), 110–133.
Park, Y., & Jo, I.-H. (2019b). Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development, 67, 1547–1571. https://doi.org/10.1007/s11423-019-09693-0
Raj, N. S., & Renumol, V. G. (2022). Early prediction of student engagement in virtual learning environments using machine learning techniques. E-Learning and Digital Media, 19(6), 537–554. https://doi.org/10.1177/20427530221108027
Ramaswami, G., Susnjak, T., & Mathrani, A. (2023). Effectiveness of a Learning Analytics Dashboard for Increasing Student Engagement Levels. Journal of Learning Analytics, 10(3), 115–134. https://doi.org/10.18608/jla.2023.7935
Reid, D., & Drysdale, T. (2024). Student-facing learning analytics dashboard for remote lab practical work. IEEE Transactions on Learning Technologies, 17, 1037–1050. https://doi.org/10.1109/TLT.2024.3354128
Rets, I., Herodotou, C., Bayer, V., Hlosta, M., & Rienties, B. (2021). Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students. International Journal of Educational Technology in Higher Education, 18, 46. https://doi.org/10.1186/s41239-021-00284-9
Saqr, M., & López-Pernas, S. (2024). Why explainable AI may not be enough: predictions and mispredictions in decision making in education. Smart Learning Environments, 11, 52. https://doi.org/10.1186/s40561-024-00343-4
Sauro, J., & Lewis, J. R. (2016). Quantifying the User Experience: Practical Statistics for User Research (2nd ed.). Morgan Kaufmann. https://www.sciencedirect.com/book/monograph/9780128023082/quantifying-the-user-experience
Shapiro, R. B., Fiebrink, R., & Norvig, P. (2018). How Machine Learning Impacts the Undergraduate Computing Curriculum. Communications of the ACM, 61(11), 27–29. https://cacm.acm.org/magazines/2018/11/232197-how-machine-learning-impacts-the-undergraduate-computing-curriculum
Shin, W. (2022). Parental Digital Literacy and Mediation.
Skulmowski, A., & Xu, K. M. (2022). Understanding Cognitive Load in Digital and Online Learning: a New Perspective on Extraneous Cognitive Load. Educational Psychology Review, 34, 171–196. https://doi.org/10.1007/s10648-021-09624-7
Subramonyam, H., & Hullman, J. (2023). Are we closing the loop yet? gaps in the generalizability of vis4ml research. IEEE Transactions on Visualization and Computer Graphics, 30(1), 672–682.
Suhirman, L., Nugroho, A. Y., Dhyanaputri, I. G. A. S., Adawiyah, R., Soraya, A., Dzaky, M. I., Alunaza, H., Akbar, S. A., Hafizi, M. Z., & Junida, D. S. (2026). Dasar Metodologi Penelitian. CV. Askara Sastra Media.
Susnjak, T., Ramaswami, G. S., & Mathrani, A. (2022). Learning analytics dashboard: a tool for providing actionable insights to learners. International Journal of Educational Technology in Higher Education, 19, 12. https://doi.org/10.1186/s41239-021-00313-7
Van Poecke, A., Tabari, H., & Hellinckx, P. (2024). Unveiling the backbone of the renewable energy forecasting process: Exploring direct and indirect methods and their applications. Energy Reports, 11, 544–557. https://doi.org/10.1016/j.egyr.2023.12.031
Vlachogianni, P., & Tselios, N. (2022). Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review. Journal of Research on Technology in Education, 54(3), 392–409.
Wang, J., Liu, S., & Zhang, W. (2024). Visual analytics for machine learning: A data perspective survey. IEEE Transactions on Visualization and Computer Graphics, 30(12), 7637–7656.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Diah Aryani, Jefry Sunupurwa Asri , Habibullah Akbar, Muhammad Fikri, Bayu Sulistiyanto Ipung Sutejo (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
.png)
