Development and Validation of a Rubric for Evaluating Online Courses
DOI:
https://doi.org/10.65168/bs.226-18Keywords:
Rubric, validation, internal consistency, inter-rater reliability, online courseAbstract
Although online learning has become increasingly prevalent in higher education in recent years, a unified framework and validated instruments for evaluating the quality of online courses remain insufficient in Mongolia. This study aimed to develop a rubric for assessing online course quality within the context of Mongolian higher education and to examine its reliability. In the first phase, 11 internationally recognized instruments for evaluating online learning were compared through document analysis to identify their common criteria, theoretical foundations, and structural characteristics. Based on these findings, a rubric comprising 6 criteria and 24 indicators was developed, grounded in the theories of Constructive Alignment, Community of Inquiry, and Universal Design for Learning. In the second phase, the rubric was piloted using 16 online courses offered by Mongolian universities. Internal consistency was assessed using Cronbach’s alpha (α), while inter-rater reliability was estimated using the intraclass correlation coefficient (ICC). The results indicated high internal consistency (α = .956) and strong inter-rater reliability (average ICC(2,2) = .880, 95% CI [.773, .950], p < .001), demonstrating that the rubric can reliably evaluate online courses within the Mongolian higher education context. The findings provide an initial theoretically grounded rubric for evaluating online course quality in Mongolian universities. However, further validation using larger samples and more comprehensive validity analyses is recommended.
References
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing. Longman.
Baldwin, S. J., Ching, Y.-H., & Hsu, Y.-C. (2018). Online course design in higher education: A review of national and statewide evaluation instruments. TechTrends, 62(1), 46–57. https://doi.org/10.1007/s11528-017-0215-z
Biggs, J. (2003). Teaching for quality learning at university (2nd ed.). Open University Press. https://www.mheducation.co.uk/teaching-for-quality-learning-at-university-9780335211746-emea-group
Булгамаа Б., Амартайван Ц., Түмэнбаяр Д. (2025). Хичээлийн видео анализад үндэслэн багшийн ур чадварыг үнэлэх рубрикийн найдвартай байдлын судалгаа. Боловсрол судлал, 27(632), 44–57.
CAST. (2018). Universal design for learning guidelines version 2.2. http://udlguidelines.cast.org
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment. The Internet and Higher Education, 2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference (4th ed.). Allyn & Bacon.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Legon, R., & Garrett, R. (2018). The changing landscape of online education (CHLOE 2). Quality Matters. https://www.qualitymatters.org/qa-resources/resource-center/articles-resources/CHLOE-2-report-2018
Lkhagvasuren, M., Chuluunbat, M., Tsendsuren, N., & Batsuuri, B. (2024). Analysis of e-learning materials in pre-service teacher training: In case of the Mongolian National University of Education (MNUE). In Proceedings of the Conference on Quality Assurance in Higher Education: Transforming education—new generation of learners (QAHE 2023) (pp. 80–88). Atlantis Press. https://doi.org/10.2991/978-94-6463-382-5_12
Мягмаргармаа, Ч., & Мөнхтуяа, Л. (2026). Суралцахуйг дэмжих хиймэл оюуны чатботын туршилт, үр дүн. Боловсрол судлал, 215(7). https://bolovsrolsudlal.mn/index.php/home/article/view/376
Martin, F., Ritzhaupt, A., Kumar, S., & Budhrani, K. (2019). Award-winning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. Online Learning, 23(1), 132-157. https://doi.org/10.24059/olj.v23i1.1329
Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Moore, M. G., & Anderson, W. G. (Eds.). (2003). Handbook of distance education. Lawrence Erlbaum Associates.
Нямсүрэн, Ц., Мөнхтуяа, Л., Бадамсүрэн, Б. (2026). Онлайн сургалтын үнэлгээний хэрэгслүүдийн талаарх тойм судалгаа. Боловсролын судалгааны Монголын сэтгүүл, 31(1). https://doi.org/10.56380/mjer.v31.1.2
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
Ratcliff, J. L., Lubinescu, E. S., & Gaffney, M. A. (Eds.). (2001). How accreditation influences assessment. New Directions for Higher Education, 113. https://doi.org/10.1002/he.1
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(1). https://doi.org/10.7275/96jp-xz07
Tudevdagva, U. (2020). Structure-oriented evaluation: An evaluation approach for complex processes and systems. Springer. https://doi.org/10.1007/978-3-030-44806-6
Tudevdagva, U., Sodnom, B., & Erdenechimeg, S. (2021). The evaluation case study of online course during pandemic period in Mongolia. Communications in Computer and Information Science, 1448, 508–532. https://doi.org/10.1007/978-3-030-87905-6_51
UNESCO. (2023). Global education monitoring report 2023. UNESCO Publishing. https://digitallibrary.un.org/record/4020460
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2020). Systematic review of research on artificial intelligence applications in higher education. International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0
