Visit Modern Campus

Challenges and Solutions of Research into the Quality of Online Learning: A Quality Matters Lens

The EvoLLLution | Challenges and Solutions of Research into the Quality of Online Learning: A Quality Matters Lens
Research is essential for refining processes and improving the quality of education, but the structure of the research project is as important as the question itself.

The existing literature showcases a vast amount of research on quality in online learning. Studies abound, diving deep into specific elements of design and delivery and their impact on various metrics of quality: retention, student engagement, student learning, student satisfaction, etc.

The Quality Matters (QM) Rubric™️ of Standards for the quality design of online and blended courses was developed based on reported best practices and educational research. Still, as part of their justification for adopting quality standards, many are eager to conduct their own research to show the gains resulting from their use of such standards. Often, this approach is fraught with pitfalls and limitations, due to problems in the research proposition, methodology and operationalization.

In this paper, we explore the challenges of educational research and report on the Quality Matters experience, concluding with considerations important in the quest for robust research.

The QM Rubric and its Supporting Research

Quality Matters originated in the early 2000s, developing a rubric of standards for the quality design of online and blended courses, professional development for implementing it, and a peer-review process to assure it. Put simply, the rubric was developed based on best practices and is supported by research. The emphasis of the rubric is on quality instructional design that supports learning.

The tenets of instructional design—defined by independent educational research—are brought together in the QM Rubric. According to the rubric’s instructional design standards, a quality online or blended course provides greater opportunity for instructors to facilitate the engagement of students, especially within the framework provided by the institution. And the research that supports and provides important details for each of the General Standards within the QM Rubric is growing daily, with hundreds of references related to each of the General Standards:

General Standard 1: Course Overview and Introduction

General Standard 2: Learning Objectives (Competencies)

General Standard 3: Assessment and Measurement

General Standard 4: Instructional Materials

General Standard 5: Learning Activities and Learner Interaction

General Standard 6: Course Technology

General Standard 7: Learner Support

General Standard 8: Accessibility and Usability

The use of QM takes many forms, such as professional development, course development and the encouragement of online student-related institutional policies. Even when QM is implemented at the optimal level, there are countless variables affecting student outcomes. A common question about the effectiveness of any process used in education is, “Are the outcomes for students improved as a result of this process?” A logical place to look for the answer is research. Yet, finding definitive answers in the body of completed research is challenging and requires weaving together what has been produced in the professional and academic literature.

The Challenges of Educational Research

While standards in the QM Higher Education Rubric are supported by research on instructional design, educators are interested in the results of actual QM implementations. They want to know how the application of the QM Rubric and process of professional development and course reviews manifest in improved outcomes for students. Finding improved student outcomes related to QM requires formal educational research that entails a long-term commitment, significant human and financial resources, a large number of participants, a large amount of data, advanced statistical expertise and—for those conducting the research—a marshaling of all of these throughout the research process. Further, formal research that checks all the boxes is often focused on a narrow topic. Measuring the Impact of the Quality Matters (QM) Rubric™️: A Discussion of Possibilities is a 2015 study that focuses on QM and illustrates the common roadblocks of conducting formal research. How Will Unresolved Research Questions Get Answered goes beyond QM in discussing the challenges of educational research.

Many times, educational research is complicated by the interaction between the intervention, the student, the delivery and the student’s environment with the result that the research shows little effect from the intervention. Nevertheless, there are a number of studies on the variety of effects QM implementation can have.

The holy grail for educational outcomes as seen by legislators and grant providers is based on residential, traditional-aged students successfully completing their program of study within six years (see Differences in Time to Degree by Age). Students also must be successfully employed in a field related to their studies within several months of completion of the program, or should be enrolled in another educational program for which the first one prepared them. This measure, in fact, omits a large percentage of postsecondary students.

Furthermore, so many variables combine to make up a student’s educational experience that it is difficult to isolate any one of them as being correlated to one or more positive impacts. There are a few possible exceptions, such as high-impact practicesand successfully earning a degree from an elite institution, but even with these correlations, it is difficult to subtract individual student characteristics such as motivation and financial support from the intervention so as to attribute the right proportion of student success to the intervention.

Considerations for Conducting Research on QM Impact

There are 10 important considerations to examine before undertaking a formal research study on the effects of educational initiatives. Here, they are given through the lens of QM:

#1 Principal’s Familiarity with QM

The researchers should have an understanding of the breadth and depth of QM tools and processes prior to taking on a study to correlate with outcome measures. Having successfully completed the Applying the Quality Matters Rubric workshop, going on to develop (and teach) courses using QM, earning the Peer Reviewer Certification, and serving as a QM-Certified Peer Reviewer gives researchers the required familiarity with QM.

#2 Access to the Necessary Data

Reliable data that will have departmental or institutional implications goes beyond what can be derived from a single course. Even after it is determined what data will answer the research question, the sources of it need to be identified (e.g., student information system, institutional research and registrar data) and mapped to the specific research.

#3 Administrative/Leadership Support

The administration or leadership should understand how the use of QM and the research on QM help achieve the organization’s goals and strategic priorities. To provide the needed information to administrators:

  • Identify and map expected outcomes to key institutional assessment needs and strategic priorities
  • Use data visualization to reaffirm the institution’s accreditation story while providing evidence and artifacts of impact
  • Align budget and staffing for collecting and analyzing data evidence

#4 Context Analysis

Researchers should assess the context of the learning environment in which QM is to be studied as an independent variable. The robust use of QM tools and processes as a new intervention is easier to track than the use of QM in an environment already impacted by it. It’s important to:

#5 Significant Use of QM 

If increasing student learning is the ultimate goal, use of QM tools and processes should be well-documented and significant rather than involving only a few courses or instructors. This gets back to the level of QM implementation. When QM is significant—that is, used supportively and systematically across academic programs or departments—the impact can be discussed along with:

  • The efforts and resource investment to improve outcomes
  • How QM use links to key institutional needs and the measures that indicate success
  • Stakeholder reactions to QM implementation and its outcomes

#6 Sample Size

While quality formal research can be done with a qualitative study or a quantitative study with a small number of students, it is difficult to draw implications from the research without having a sizable sample population and, when possible, randomization of interventions. To deal with this consideration:

  • Work with the administration for access to the broadest, most representative sample of students
  • Identify and work with data keepers in your institution to utilize existing student data

#7 Time

A study on the effect of QM is more likely to be informative if it maps and analyzes targeted implementation over a period of several semesters or—even better—years. While reflective practitioners regularly gather and analyze their own actions and courses as part of their ongoing commitment to improve their own courses and teaching, an investigation of the impact of Quality Matters tools and processes on courses requires a more deliberate research approach. To achieve this:

  • Build research relationships at your institution and beyond
  • Build data banks for future research use

#8 Statistical Expertise

Advice from someone with advanced statistical expertise is key to getting valid information and reliable analysis from the data. In educational research, isolating the impact of an independent variable—in this case, some aspect of QM implementation—is aided by propensity score matching, for which the advice or active involvement of a statistical expert is helpful. To do so:

  • Identify and establish relationships with statistical experts at your institution
  • Identify and obtain access points for needed data

#9 Pre- and Post-Testing

With any given sample population, the results of an intervention are most meaningful when compared to that population’s starting point. Pre- and post-testing of students related to the topic under investigation should be conducted to help determine whether or not the intervention worked. To get started:

  • Define and address the conditions surrounding the pre- and post-testing (see #4 above)
  • Strategize how to take advantage of pre- and post-testing, while mitigating its flaws

#10 Keeping the Research Goal in Mind  

Conducting formal research is a significant undertaking, so having a good understanding of the purpose of research is important. It’s helpful to:

  • Establish the purpose of the research project: Is it to gain detailed knowledge of the effects of targeted implementations or to evaluate specific problem-solving solutions as part of a continuous improvement plan?
  • Discuss and establish collaborative institutional support for the project
  • Integrate the data, analyses, strategic alignments and collaborative staffing relationships of a research project into a procedural and cultural emphasis on tracking and producing evidence of student learning


Educational research can be a challenge, but it is not impossible. The experience of gathering research on QM implementation over more than a decade has revealed a number of factors, such as familiarity with the intervention and contextual analysis, that can help researchers improve the learning experience.


– – – –


Barshay, J. (2018). The ‘dirty secret’ about educational innovation. The Hechinger Report. Retrieved from

Dzuiban, C., Howlan, C., Moskal, P., Johnson, C., Eid, M., & Kmetz, B. (2018). Adaptive learning: context and complexity. e-mentor. Retrieved from:

Kuh, G.D. (2008). Excerpt from High-Impact Educational Practices. Association of American Colleges & Universities. Retrieved from

Lane, F. (n.d. ). Propensity Score Matching: A Primer for Educational Researchers. Retrieved from

Legon, R. (2015). Measuring the impact of the Quality Matters Rubric™: a discussion of possibilities. The American Journal of Distance Education, 29(3), 166-173,DOI: 10.1080/08923647.2015.1058114

Lieberman, M. (2018). How will unresolved research questions get answered? Inside Higher Ed. Retrieved from

National Center for Education Statistics. (2019). Characteristics of Postsecondary Students. Retrieved from:

Quality Matters. (2019). About QM. Retrieved from

Quality Matters. (2016). Process. Retrieved from

Quality Matters. (2019). Confound Iit! Or, why it’s important not to. Retrieved from

Quality Matters. (2019). Curated Resources. Retrieved from

Quality Matters. (2019). QM Research Library. Retrieved from

Quality Matters. (2019). QM Rubrics & Standards. Retrieved from

Quality Matters. (2019). How to Conduct QM Research. Retrieved from

Quality Matters. (2016). QM Impact Summary. Retrieved from

Qualtrics. (2019). Determining Sample Size: How to Ensure You Get the Correct Sample Size. Retrieved from

Schön, Donald A. (1983). The reflective practitioner: how professionals think in action. New York: Basic Books. ISBN 978-0465068746. OCLC 8709452

Shapiro, D., Dundar, A., Wakhungu, P.K., Yuan, X., Nathan, A, & Hwang, Y. (2016, September). Time to Degree: A National View of the Time Enrolled and Elapsed for Associate and Bachelor’s Degree Earners (Signature Report No. 11). Herndon, VA: National Student Clearinghouse Research Center. Retrieved from

Value-Added Assessment (Pre- and Post-Testing). (n.d.). Skidmore College Website. Retrieved from

Author Perspective: