Launching any program from scratch is a major undertaking for a postsecondary institution, especially when it needs to be self-supporting or revenue-generating. When the program being launched is the first of its kind, with no set playbook and minimal examples to emulate, judging success and identifying the path forward is especially challenging. However, as data-driven decision making—facilitated by the collection and leveraging of analytics—becomes a more common practice, the burden on innovative administrators is eased. In this interview, Aaron Brower reflects on the critical role data has played in the launch and development of the innovative University of Wisconsin Flexible Option (UW-Flex) program.
The EvoLLLution (Evo): Broadly speaking, why are Big Data and data analytics becoming increasingly important to the management of non-traditional postsecondary organizations?
Aaron Brower (AB): Big Data and data analytics have become increasingly important across all of higher education, so let’s start there. The need to justify the value of higher education couldn’t be more important and the competition for students couldn’t be stronger. In addition, what “counts” as higher education is less clear now since different stakeholders are asking different things from institutions of higher ed: some institutions are offering badges and stackable certificates, while others are focused narrowly on job placement and filling the “skills gap.” Still others only equate higher education with liberal education.
Those of us who are leaders in higher education need to be much more deliberate about our goals for our institutions, and data analytics provides the tools to fine tune what we offer, fine tune who we’re offering educational programs to, and fine tune program effectiveness and efficiency.
The number of people seeking higher education continues to rise, with well over 70 percent of the general public now seeking some form of postsecondary education. That’s almost double the rate from 50 years ago. At the same time, costs for higher ed have risen dramatically, by some accounts at an even faster rate than healthcare costs. And for public higher ed in particular, public funds have been in decline for almost 50 years. Even though a majority of states have begun to modestly reinvest in their institutions of higher education, we will never go back to the peak of public funding that existed in the late 1970s and early 1980s. As a consequence, institutions of higher ed are being forced to be more deliberate and even mercenary about what they offer, and to whom, because an increasing share of their revenue needs to come from tuition and other non-state sources. Data analytics provides the tools to better understand, among other things, how to attract students to our programs and how our students are doing once they enroll.
And let’s make sure we know what we’re talking about here. Data analytics most often refers to the range of tools that are used to mine the mountain of information that is now generated by all sorts of online apps, programs and resources. We can track everything from clicks and entries into CRMs, LMSs and SISs, to search paths through program websites, to more standard records from surveys. “Big Data” describes this mountain of often dissimilar data that are linked together so data analytic tools can be used to search out patterns, correlations and trends across the various data elements. Ways to join dissimilar data elements, and the data analytic tools used to mine these data, have grown significantly in recent decades, driven primarily by online businesses. Higher education is just beginning to catch up. For example, think about the higher education parallels to online vendors like Amazon and Pandora, who offer product recommendations along the lines of “if you liked that, we know you’ll like this…”
Institutions of higher education are seeing the value of mining Big Data in order to better understand and predict behaviors of their students. Early examples were like Purdue’s SIGNALS project that identified which of their students were at higher risk for course failure. Uses of Big Data and data analytics now include investigations into all kinds of student behavior, from the work done by EAB and other enrollment management companies, to our own homegrown investigations into student learning patterns for the UW Flexible Option program.
Now, as for the additional relevance of using Big Data and data analytics for non-traditional students and non-traditional postsecondary institutions—this is particularly important because it’s easy to overgeneralize incorrectly by by applying what we know about traditional students to non-traditional students and non-traditional educational programs. It is almost a cliché to say that we need to understand the unique behaviors of non-traditionals rather than make assumptions based on traditional programming—on everything from how students learn, to how they access information, to their educational goals. Big Data and data analytics offer a way to investigate non-traditional students carefully and rigorously, based on their own behaviors and not as a corollary to behaviors of traditional students.
Evo: Conversely, what would you identify as a few limits to the value of Big Data in managing a postsecondary institution?
AB: The adage of garbage in/garbage out comes to mind. While Big Data has been used successfully to predict purchasing behavior, weather forecasting and urban growth, those applications all have a clear set of outcomes—whether that’s purchasing totals and items purchased, temperatures and weather patterns, or population density. On the other hand, one of the more enduring problems in educational outcome research relates to fuzzy definitions for student learning.
For example, are we satisfied when graduates are simply proficient in their disciplinary knowledge, or don’t they also need to apply their knowledge to novel (and often vague) new settings? Further, don’t we want graduates to be active and engaged citizens of the world? But what does that mean exactly? Additionally, the research literature has variously defines student success as meeting one’s own goals, as meeting goals of future employers or other stakeholders, and as learning and memorizing objective knowledge. Not only do each of these definitions have their limitations, but the definition you pick leads you down a different analytical and programmatic path.
It is critical, in other words, to be clear about the questions that program planners are trying to answer through Big Data and predictive analytics. Understanding and prioritizing one’s own question is the first important task for anyone who wants to use Big Data and data analytics in higher education.
Another limitation exists on the use of Big Data and data analytics, too. “Big Data” is seductive because it feels like one can use micro behaviors to “see” into student decision making. But “Big Data” requires two things to make it work: enough data so that patterns found are not anecdotal, and skilled analysts who are able to look beyond the noise to see real and enduring patterns. Data analytics tools are still being developed within the domain of education research, and the pool of people who can effectively use these tools is still relatively small. In fact, one of the collaborative degree programs that we offer, our Masters in Data Science, has been wildly successful since its inception because of the shortage of these kinds of programs available.
Evo: In the management of the UW Flex program, what are some of the most important sets of data you and your team are tracking?
AB: The current set of questions that are top of mind for us are
1. Are students learning what they should be learning? (Which itself has two sub-questions):
a) Have we identified the right learning outcomes based on the professionals in the UW Flex fields of study?
b) Are students achieving those outcomes?
2. Is UW Flex attracting the students to our programs that we intend to attract, and are we serving them well?
3. Are we spending our money where it is most needed to maintain high-quality UW Flex programs that are meeting the needs of our students and stakeholders?
Then, to answer these questions,
1. We need outcome data about student learning—are students achieving competency on the various projects and assessments that constitute our programs?
2. We need input data about who our students are.
3. And we need throughput data—throughout the student lifecycle, from recruitment to graduation—about how our program is accessed and used.
We gather these data primarily from our CRM, LMS and SIS, and we also use post-graduation surveys to capture information about graduates. Additionally, we use focus groups and interviews to keep in routine contact with employers and other stakeholders. We do not manage all of these data in one giant database, but we are working towards that goal.
Evo: What are a few examples of ways you and your colleagues have leveraged the information gleaned from data analytics to improve the experience and success of UW-Flex students?
AB: We evaluate the enrollment funnel very carefully. We want to make sure we’re serving our primary audience well, so we not only evaluate how best to recruit prospective students, but we continually survey our target audience to make sure we’re offering what is in demand. We also use post-graduation surveys that capture students’ experiences and the experiences of their employers.
On the continuous-program-development front, UW Flex uses data from our LMS to evaluate how students are responding to each of our competency projects and assessments. What do patterns of completion look like? Are some projects and assessments too easy or too difficult? Is the feedback received from faculty pitched correctly to urge students forward? Likewise, is support given by our Academic Success Coaches enough or too much, and in the right domains?
One more example: While identifying a new definition for retention, we also discovered a pattern in our enrollment data that we are now using to improve student retention and success rates. When non-traditional students, like those in UW Flex, are given the option to start and stop their school work with no penalty, we see that retention is strongly correlated to early program success. Moreover, the positive impact of this success persisted even when that student was not successful later on.
This finding helped us look carefully at how the projects and assessments within each of our Flex programs were sequenced, as well as the types of supports we gave students, particularly at the beginning. We now use these results to “calibrate” our assessments and our support to find the sweet spot that best helps students without coddling them. More information about our analyses and findings appear in a paper written for the Journal of Competency-Based Education, “Student Success and Retention Using New Definitions Created for Non-Term, Direct Assessment CBE,” published in February 2017.
Evo: In an earlier piece in the Navigating the CBE Frontier series, Managing Enrollment and Revenue Targets, you outlined how you and your colleagues used enrollment trends analyses to model anticipated revenue and expenses for UW Flex. How accurate did those models end up being, and how have they evolved over time?
AB: UW Flex is built as a cost-recovery program, making us dependent on enrollments to support ongoing program expenses. At the same time, the University of Wisconsin is a public nonprofit institution, so we seek only enough program revenue to break even. Suffice it to say that setting and achieving enrollment targets is critical to the success of UW Flex.
Our predictive models have improved with time and experience. Our initial enrollment targets were based on experiences with students in our traditional online programs. Student behavior through UW Flex is different, and we get better with each new year of experience.
Just recently, we reevaluated and fine-tuned our enrollment targets based on a nuance observed in student enrollment behavior: New students and continuing students exhibit a slightly different pattern of enrollment. As mentioned previously, we found that new students, if they have a success early in their experience with us, will persist at much higher rates, even if they have subsequent failures. To become continuing students, in other words, they needed an “early win.” And once students become a continuing student—in our case, when they reenroll in their UW Flex program within six months—their retention rates are very high, regardless of their pace through the remainder of their program. Understanding our students’ enrollments with this level of detail has allowed us to substantially improve our enrollment targets.
And we know we can improve our predictions even more. Different students, for example, achieved their early success differently—some by taking the UW Flex equivalent of general education courses, and some by diving directly into a disciplinary specialty area (such as diving head first into an IT coding project for some of our IT students). We can’t yet predict which students will find early success from general education or disciplinary specialization, but this is the next set of questions that we are pursuing.
Big Data requires enough data so that patterns are stable enough to justify their use in program decisions. As mentioned, our models become better with each new student and each additional year of UW Flex. Ultimately, data analytics will allow us to zero in on the behavior of finer and finer subsets of our students, which will allow us to develop more sophisticated and fine-grained strategies to maximize our students’ abilities to succeed.
To learn more about the work UW-Extension is doing to grow UW-Flex, please click here.
You Might Also Like
