The existing literature showcases a vast amount of research on quality in online learning. Studies abound, diving deep into specific elements of design and delivery and their impact on various metrics of quality: retention, student engagement, student learning, student satisfaction, etc.
The Quality Matters (QM) Rubric™️ of Standards for the quality design of online and blended courses was developed based on reported best practices and educational research. Still, as part of their justification for adopting quality standards, many are eager to conduct their own research to show the gains resulting from their use of such standards. Often, this approach is fraught with pitfalls and limitations, due to problems in the research proposition, methodology and operationalization.
In this paper, we explore the challenges of educational research and report on the Quality Matters experience, concluding with considerations important in the quest for robust research.
The QM Rubric and its Supporting Research
Quality Matters originated in the early 2000s, developing a rubric of standards for the quality design of online and blended courses, professional development for implementing it, and a peer-review process to assure it. Put simply, the rubric was developed based on best practices and is supported by research. The emphasis of the rubric is on quality instructional design that supports learning.
The tenets of instructional design—defined by independent educational research—are brought together in the QM Rubric. According to the rubric’s instructional design standards, a quality online or blended course provides greater opportunity for instructors to facilitate the engagement of students, especially within the framework provided by the institution. And the research that supports and provides important details for each of the General Standards within the QM Rubric is growing daily, with hundreds of references related to each of the General Standards:
General Standard 1: Course Overview and Introduction
General Standard 2: Learning Objectives (Competencies)
General Standard 3: Assessment and Measurement
General Standard 4: Instructional Materials
General Standard 5: Learning Activities and Learner Interaction
General Standard 6: Course Technology
General Standard 7: Learner Support
General Standard 8: Accessibility and Usability
The use of QM takes many forms, such as professional development, course development and the encouragement of online student-related institutional policies. Even when QM is implemented at the optimal level, there are countless variables affecting student outcomes. A common question about the effectiveness of any process used in education is, “Are the outcomes for students improved as a result of this process?” A logical place to look for the answer is research. Yet, finding definitive answers in the body of completed research is challenging and requires weaving together what has been produced in the professional and academic literature.
The Challenges of Educational Research
While standards in the QM Higher Education Rubric are supported by research on instructional design, educators are interested in the results of actual QM implementations. They want to know how the application of the QM Rubric and process of professional development and course reviews manifest in improved outcomes for students. Finding improved student outcomes related to QM requires formal educational research that entails a long-term commitment, significant human and financial resources, a large number of participants, a large amount of data, advanced statistical expertise and—for those conducting the research—a marshaling of all of these throughout the research process. Further, formal research that checks all the boxes is often focused on a narrow topic. Measuring the Impact of the Quality Matters (QM) Rubric™️: A Discussion of Possibilities is a 2015 study that focuses on QM and illustrates the common roadblocks of conducting formal research. How Will Unresolved Research Questions Get Answered goes beyond QM in discussing the challenges of educational research.
Many times, educational research is complicated by the interaction between the intervention, the student, the delivery and the student’s environment with the result that the research shows little effect from the intervention. Nevertheless, there are a number of studies on the variety of effects QM implementation can have.
The holy grail for educational outcomes as seen by legislators and grant providers is based on residential, traditional-aged students successfully completing their program of study within six years (see Differences in Time to Degree by Age). Students also must be successfully employed in a field related to their studies within several months of completion of the program, or should be enrolled in another educational program for which the first one prepared them. This measure, in fact, omits a large percentage of postsecondary students.
Furthermore, so many variables combine to make up a student’s educational experience that it is difficult to isolate any one of them as being correlated to one or more positive impacts. There are a few possible exceptions, such as high-impact practicesand successfully earning a degree from an elite institution, but even with these correlations, it is difficult to subtract individual student characteristics such as motivation and financial support from the intervention so as to attribute the right proportion of student success to the intervention.
Considerations for Conducting Research on QM Impact
There are 10 important considerations to examine before undertaking a formal research study on the effects of educational initiatives. Here, they are given through the lens of QM:
#1 Principal’s Familiarity with QM
The researchers should have an understanding of the breadth and depth of QM tools and processes prior to taking on a study to correlate with outcome measures. Having successfully completed the Applying the Quality Matters Rubric workshop, going on to develop (and teach) courses using QM, earning the Peer Reviewer Certification, and serving as a QM-Certified Peer Reviewer gives researchers the required familiarity with QM.
#2 Access to the Necessary Data
Reliable data that will have departmental or institutional implications goes beyond what can be derived from a single course. Even after it is determined what data will answer the research question, the sources of it need to be identified (e.g., student information system, institutional research and registrar data) and mapped to the specific research.
#3 Administrative/Leadership Support
The administration or leadership should understand how the use of QM and the research on QM help achieve the organization’s goals and strategic priorities. To provide the needed information to administrators:
- Identify and map expected outcomes to key institutional assessment needs and strategic priorities
- Use data visualization to reaffirm the institution’s accreditation story while providing evidence and artifacts of impact
- Align budget and staffing for collecting and analyzing data evidence
#4 Context Analysis
Researchers should assess the context of the learning environment in which QM is to be studied as an independent variable. The robust use of QM tools and processes as a new intervention is easier to track than the use of QM in an environment already impacted by it. It’s important to:
- Account for the variety of features that impact quality online courses and the level of QM implementation
- Identify the input and output variables relevant to what is being measured, including the independent and dependent variables
- Create spreadsheets to input data for analysis, and analyze the data according to how it fits with the research question and with institutional strategies and goals
#5 Significant Use of QM
If increasing student learning is the ultimate goal, use of QM tools and processes should be well-documented and significant rather than involving only a few courses or instructors. This gets back to the level of QM implementation. When QM is significant—that is, used supportively and systematically across academic programs or departments—the impact can be discussed along with:
- The efforts and resource investment to improve outcomes
- How QM use links to key institutional needs and the measures that indicate success
- Stakeholder reactions to QM implementation and its outcomes
#6 Sample Size
While quality formal research can be done with a qualitative study or a quantitative study with a small number of students, it is difficult to draw implications from the research without having a sizable sample population and, when possible, randomization of interventions. To deal with this consideration:
- Work with the administration for access to the broadest, most representative sample of students
- Identify and work with data keepers in your institution to utilize existing student data
#7 Time
A study on the effect of QM is more likely to be informative if it maps and analyzes targeted implementation over a period of several semesters or—even better—years. While reflective practitioners regularly gather and analyze their own actions and courses as part of their ongoing commitment to improve their own courses and teaching, an investigation of the impact of Quality Matters tools and processes on courses requires a more deliberate research approach. To achieve this:
- Build research relationships at your institution and beyond
- Build data banks for future research use
#8 Statistical Expertise
Advice from someone with advanced statistical expertise is key to getting valid information and reliable analysis from the data. In educational research, isolating the impact of an independent variable—in this case, some aspect of QM implementation—is aided by propensity score matching, for which the advice or active involvement of a statistical expert is helpful. To do so:
- Identify and establish relationships with statistical experts at your institution
- Identify and obtain access points for needed data
#9 Pre- and Post-Testing
With any given sample population, the results of an intervention are most meaningful when compared to that population’s starting point. Pre- and post-testing of students related to the topic under investigation should be conducted to help determine whether or not the intervention worked. To get started:
- Define and address the conditions surrounding the pre- and post-testing (see #4 above)
- Strategize how to take advantage of pre- and post-testing, while mitigating its flaws
#10 Keeping the Research Goal in Mind
Conducting formal research is a significant undertaking, so having a good understanding of the purpose of research is important. It’s helpful to:
- Establish the purpose of the research project: Is it to gain detailed knowledge of the effects of targeted implementations or to evaluate specific problem-solving solutions as part of a continuous improvement plan?
- Discuss and establish collaborative institutional support for the project
- Integrate the data, analyses, strategic alignments and collaborative staffing relationships of a research project into a procedural and cultural emphasis on tracking and producing evidence of student learning
Conclusion
Educational research can be a challenge, but it is not impossible. The experience of gathering research on QM implementation over more than a decade has revealed a number of factors, such as familiarity with the intervention and contextual analysis, that can help researchers improve the learning experience.
– – – –
References
Barshay, J. (2018). The ‘dirty secret’ about educational innovation. The Hechinger Report. Retrieved from https://hechingerreport.org/the-dirty-secret-about-educational-innovation/?utm_source=The+Hechinger+Report&utm_campaign=13d2040785-EMAIL_CAMPAIGN_2018_12_14_06_26&utm_medium=email&utm_term=0_d3ee4c3e04-13d2040785-322711453
Dzuiban, C., Howlan, C., Moskal, P., Johnson, C., Eid, M., & Kmetz, B. (2018). Adaptive learning: context and complexity. e-mentor. Retrieved from: http://www.e-mentor.edu.pl/eng/article/index/number/77/id/1384
Kuh, G.D. (2008). Excerpt from High-Impact Educational Practices. Association of American Colleges & Universities. Retrieved from https://www.aacu.org/leap/hips
Lane, F. (n.d. ). Propensity Score Matching: A Primer for Educational Researchers. Retrieved from https://slideplayer.com/slide/6250920/
Legon, R. (2015). Measuring the impact of the Quality Matters Rubric™: a discussion of possibilities. The American Journal of Distance Education, 29(3), 166-173,DOI: 10.1080/08923647.2015.1058114
Lieberman, M. (2018). How will unresolved research questions get answered? Inside Higher Ed. Retrieved from https://www.insidehighered.com/digital-learning/article/2018/11/07/research-digital-learning-faces-uncertain-future-funding-sources#.XAfXCJk2LD8.email
National Center for Education Statistics. (2019). Characteristics of Postsecondary Students. Retrieved from: https://nces.ed.gov/programs/coe/indicator_csb.asp
Quality Matters. (2019). About QM. Retrieved from https://www.qualitymatters.org/why-quality-matters/about-qm
Quality Matters. (2016). Process. Retrieved from https://www.qualitymatters.org/why-quality-matters/process
Quality Matters. (2019). Confound Iit! Or, why it’s important not to. Retrieved from https://www.qualitymatters.org/qa-resources/resource-center/articles-resources/confounding-variables-in-research
Quality Matters. (2019). Curated Resources. Retrieved from https://www.qualitymatters.org/research/curated-research-resources
Quality Matters. (2019). QM Research Library. Retrieved from https://www.qmprogram.org/qmresources/research/
Quality Matters. (2019). QM Rubrics & Standards. Retrieved from https://www.qualitymatters.org/qa-resources/rubric-standards
Quality Matters. (2019). How to Conduct QM Research. Retrieved from https://www.qualitymatters.org/research/conduct-qm-research
Quality Matters. (2016). QM Impact Summary. Retrieved from https://www.qualitymatters.org/qa-resources/resource-center/articles-resources/qm-impact-summary
Qualtrics. (2019). Determining Sample Size: How to Ensure You Get the Correct Sample Size. Retrieved from https://www.qualtrics.com/experience-management/research/determine-sample-size/
Schön, Donald A. (1983). The reflective practitioner: how professionals think in action. New York: Basic Books. ISBN 978-0465068746. OCLC 8709452
Shapiro, D., Dundar, A., Wakhungu, P.K., Yuan, X., Nathan, A, & Hwang, Y. (2016, September). Time to Degree: A National View of the Time Enrolled and Elapsed for Associate and Bachelor’s Degree Earners (Signature Report No. 11). Herndon, VA: National Student Clearinghouse Research Center. Retrieved from https://nscresearchcenter.org/signaturereport11/
Value-Added Assessment (Pre- and Post-Testing). (n.d.). Skidmore College Website. Retrieved from https://www.skidmore.edu/assessment/archived/pre-or-post-assessment.php
You Might Also Like
