This article borrows heavily from “Measuring What Matters: UW Flexible Option’s New Path: Measuring Success from a Student Vantage Point,” a chapter written for the soon-to-be-published book, “Handbook of Research on Competency-Based Education in University Settings.”[1] We wrote this chapter through a grant from Lumina Foundation, who has supported our ability to tell our story and establish evaluation metrics.
This post will only provide the briefest of introductions to our evaluation framework. Please refer to the chapter referenced above for a complete description of our framework and the metrics.
One ancillary benefit to a CBE approach is that student and program success can focus directly on student learning, not proxies for student learning. Universities that use traditional educational models based on credits and semesters measure their success through persistence and graduation rates, credits earned, the pace at which students accrue credits, and grades on courses. As is well established, seat time (even fast-paced or efficient seat time) is not a good measure of student learning.
On the other hand, for CBE programs—and especially direct-assessment CBE programs such as the University of Wisconsin Flexible Option (UW Flex)—students only make progress when they demonstrate that they have achieved mastery over pre-determined skills, abilities and knowledge. This article will briefly describe the framework we are developing to measure student and program success. Some aspects of the UW Flex metrics framework are developed and in use while others are still in development. However, I will describe our full model in order to convey the intent and direction that is guiding our evaluation.
The UW Flex Metrics Framework: Three Student-Level and Three Program-Level Measures
Students have greater scheduling and learning flexibilities in UW Flex’s non-credit, non-semester model. Students enroll for three-month subscription periods, starting any month of the year. Within this all-you-can-learn model, students receive customized academic support and have full access to all learning materials that allow them to progress at their own pace.
It was important for us to measure student and program success using metrics that capture actual student learning as well as individual variance in other aspects of the student experience—such as students’ expectations and goals for their own learning.
We set as a principle that program success should aggregate from student-level metrics. Our framework, therefore, consisted of six variables: three individual-level variables of academic outcomes, pace, and goals, and each of these three aggregated to the program level.
By aggregating student pace, outcomes and goals to the program level, we are able to evaluate whether students, in the aggregate, are meeting the expectations of internal and external stakeholders (e.g., business plan/budgetary goals, faculty, disciplinary and degree requirements, accreditor expectations, etc.).[2] Continual program development can be based on these program-level measures.
The figure below illustrates the components of the UW Flex metrics framework, built around the three essential metrics of goals, pace and academic outcomes.
Understanding the Six Metrics
1. Student-Level Educational Goals
This metric measures the reasons students enroll in UW Flex, whether those goals have been met, and whether the student is satisfied with their educational experience.
It is important to capture individual student goals, in order to understand the student’s progress and experience. One nursing student, for example, might be interested in earning a BSN while another might primarily be interested in developing leadership skills. Goals are captured during the development of the student’s Individualized Learning Plan (ILP), by asking what the student hopes to gain through UW Flex. Satisfaction with the educational experience will be measured through survey questions about students’ ability to attain their educational goals and their satisfaction with various aspects of their education with UW Flex. Additionally, we know that students’ goals can change throughout their enrollment; we therefore capture these changes each time their ILP is reestablished at the beginning of each three-month subscription period.
2. Student-Level Educational Pace
This metric includes three parts. First, we document the rate at which a student completes their assessments within each subscription period. Second, we document that pace relative to the student’s planned rate of completion (captured in the ILP). Third, we assess the nature of the student’s engagement with the curriculum. Measurement of a student’s engagement with the curriculum resulted from our application to the US Department of Education to award Title IV financial aid. We needed to demonstrate that students engaged in “meaningful work” within the first 10 days of their subscription period, and we did so by creating a Goal Setting Assignment that students complete within the first 10 days of each subscription period, in which they reestablish and rededicate themselves to their ILP.
3. Student-Level Academic Outcomes
Measuring student-level academic outcomes is straightforward: It captures which competencies the student masters and completes within each three-month subscription period. Additionally, we will also ask students their view of the value and impact of mastered competencies and UW Flex.[3]
4. Program-Level Educational Goals (aggregating student-level goals)
At the program level, we look for patterns among student-level goals, alignment with program goals, levels of satisfaction, and goals by student type. For example, (and continuing our example of different students enrolled in the UW Flex BSN):
- What proportion of students are interested in achieving the Bachelor of Science in Nursing vs. individual competency areas (such as leadership)?
- Which competency areas are most sought out? Are there patterns among the students who have different goals? For example, are students pursuing the Bachelor of Science in Nursing more likely to have been in their job for more than seven years or to be seeking a promotion?
- Are there demographic or professional-interest profiles of students who have certain educational goals within a program?
- Are students who pursue certain competencies more (or less) satisfied with their education than others?
5. Program-Level Educational Pace (aggregating student-level pace)
Aggregating student-level pace allows us to examine patterns of student pace for each program. How long do students take to complete their program? Or, on which parts of the program do students proceed faster or slower? Are there patterns of student pace for students with different backgrounds, or based on their goals or life circumstances? By aggregating across student-level pace, we are able to evaluate the success of programs based on demographic and professional-interest profiles, determining which students are moving faster and slower, and through which parts of the program.
Retention and graduation rate will take on additional meaning within the context of how we measure student pace at the program level. Some students will progress much more quickly than their brick-and-mortar student peers, while others will move in and out of our programs as their lives allow. Program-level average pace, therefore, must be evaluated against students’ expectations and satisfaction with their individual pace.
6. Academic Outcomes (aggregating student-level pace)
This metric is straightforward—at its core we are measuring what students are learning across a program. In addition, we will be able to examine other elements of student learning. For example, which program competencies are mastered on their first attempt, and which take multiple attempts? Do students progress through competencies using predictable paths, either overall or correlated to prior experiences and backgrounds? Are employers and graduate schools satisfied with UW Flex graduates?
CONCLUSION
Again, the promise of CBE programs, and particularly direct-assessment CBE programs like UW Flex, is that they measure actual student learning vs. proxies of student learning. For the evaluation framework of UW Flex, we honor this promise, and build our evaluation framework around student-level information. Our evaluation framework is built to capture the granular assessment of learning within the goals and intended pace of individual students as well as across an entire program.
Operationalizing our metrics framework continues as a work in progress. And challenges abound—from the creation of new evaluation tools, to the records system used to capture data, to automating the statistics used to crunch the data, to “playing nice” with the federal and accrediting regulatory environment in which we live. (We enumerate many of these challenges in the chapter on which this blog post is based.) Suffice it to say, and like all aspects of UW Flex, we believe our idealistic efforts will be well worth it in the end.
– – – –
References and Endnotes
[1] Brower, A.M., Kallio, S., Karoff, R., Mailloux, M., Schejbal, D. “Measuring What Matters: UW Flexible Option’s New Path: Measuring Success from a Student Vantage Point”. From the Handbook of Research on Competency-Based Education in University Settings, edited by Karen Rasmussen, Pamela Northrup and Robin Colson. Copyright 2017, IGI Global. Reprinted by permission of the publisher.
[2] We also capture program-level metrics aligned with IPEDS; we need to provide these data for state and federal reporting (including to be eligible to award Title IV financial aid) and to our regional accreditor. However, this post and the chapter from which it comes focuses on the primary innovations in our measurement framework, which are the six variables described.
[3] As I have described in other posts about UW Flex, the quality of UW Flex comes from using the same UW faculty who teach in the UW System’s traditional online and brick-and-mortar programs, and from the integrity of the UW Flex curricula within specific programs. We use backward design principles when designing the curriculum, starting with the full range of skills, abilities and knowledge required by a graduate of the program (i.e., the skills, abilities and knowledge required to be a bachelor’s-level nurse, for example), which determine authentic assessments that evaluate mastery and application of those skills, abilities and knowledge, and ending with the resource that we provide to students so they can tackle their assessments. See Wiggins, G. & McTighe, J. (2005). Understanding by Design, 2nd Ed. Prentice Hall.
To learn more about the work UW-Extension is doing to grow UW-Flex, please click here.
You Might Also Like
