The following is a Q&A conducted with Scott McLean, who has been the Director of Continuing Education at the University of Calgary since 2005. A recognized expert in the field, McLean has experience teaching graduate courses in adult education theory and practice, and has practical experience ranging from teaching adult basic education in Nunavut to developing innovative continuing education programming in agricultural leadership and health promotion. In this Q&A, McLean discusses methods of measuring the success of non-degree/credit and non-traditional programming.
1. How does the University of Calgary define non-traditional education differently?
There are some universities in Canada for whom continuing education really represents the provision of degree and for-credit studies by means other than traditional daytime, on-campus, face-to-face instruction. Some universities’ continuing education operations are largely about either evening programs, or online—but the courses are very similar in their content to the normal or traditional classes, the difference is in their format or their scheduling.
…The students in continuing education programs are more likely to be part-time students. They are more likely to be mature or adult students who already have full-time jobs or careers. So one approach to looking at non-traditional programming in Canadian universities is to say “how do we deliver our degree/credit programs at times or through media which are somehow not conventional, which are different from what we deliver during the daytime?”
From our perspective—and many universities in Western Canada share the University of Calgary’s characteristic in this regard—we do some of that work, however for us continuing education also involves a very significant effort to reach out to students who are not studying for their degrees.
One example would be corporate or customized training programs; where one organization identifies it has certain professional development needs for members of their staff and comes to the university to help meet those needs.
A second example could be the provision of courses in the evenings which could be of general interest, like learning a new language or learning about the history of an event or country. They may even be professionally- or location-oriented, which means they may have an interest in taking courses in management or in relation to their profession, but not following for a degree.
2. How do practitioners of this particular style of non-traditional education determine success?
When I’m trying to determine how to determine the success of non-traditional programs, I’m looking at programs which are not leading to degrees.
So these programs could be corporate training, they could be offered to the general public or to members of particular professions—but they’re characterized by being more short-term in nature and not being characterized by the traditional courses that lead to degrees.
Each program has unique goals and it would be difficult to surmise how to actually evaluate the success of a wide range of programs because they have very different goals. Some programs may be connected to professional development for students who are practicing professionals in a particular field. Others may be opportunity outreach programs that have more social goals and others may be of a liberal arts nature where the goals simply is to broaden the minds or enrich the knowledge of the participants. …
There are reasonably standard goals against which many non-degree programs or non-credit programs could be evaluated, and three would be very simple.
One would be quality; was the actual teaching and learning of high quality?
The second thing indicator or set of goals would be that of demand; did anybody show up? Did the educational program or event attract enough interest so that there were… enough learners taking place? Did the program meet a significant demand for learning?
The third goal would be a financial one. I would say at virtually every university in North America these days, non-degree programming is expected to either be cost-recovering or revenue-generating. So the third and very important way to evaluate the success of a non-degree/credit program is did it generate enough revenues to either (a) cover all of its direct/indirect costs, or (b) generate a positive net revenue to contribute towards the other needs of the continuing education unit or the university as a whole.
Those would be the three basic categories against which we could evaluate non-degree/credit programs; quality, demand and financial return.
3. Is there a process or workflow in place to evaluate non-degree/credit programs?
The most difficult one is to assess the quality of the teaching and learning which took place, and that’s not distinct to non-credit/degree programming…
In the area of corporate training or human resource development there is a popular framework that’s in use quite broadly and it’s named for an American by the name of Donald Kirkpatrick, and it’s simply known as the Kirkpatrick framework or the Kirkpatrick Model.
It’s a very simple, very practical approach to evaluating what I would consider to be the quality of an educational activity or program… What the Kirkpatrick framework says is that there are four stages or levels at which you need to evaluate education or training activities. The first level is the level of reaction or satisfaction of students. In other words, how did the participants respond to the activity or program? Were they happy with it? Were they satisfied with it? Satisfaction is a very basic level of measuring the response of the learners or participants.
Of course, we want to move beyond that level; just having happy campers isn’t typically enough for educators, we want people who’ve learned something. So the second level of the Kirkpatrick framework is learning, and there we’re looking at changes in knowledge, skills or attitudes that have taken place as a result of the educational or training activities that the participant took part in. So it’s important for us to evaluate the extent to which people have actually learned, developed new skills, from their educational or training experience.
In some cases, when the investment in the program or activity has been high enough, or where the stakeholders involved… has a strong enough interest in the outcomes of the training program/educational program—the third level of the Kirkpatrick framework is behavior change. …
We’re not simply saying “Have they learned something?”, but as a result of learning something, have they actually changed something about either their professional practice—if it’s a professional development program—or something about the way that they are living if it’s a general interest or liberal studies program.
The final and fourth level in the Kirkpatrick framework looks at results, or it’s also called “impact”. It’s one thing for a trainee or someone who’s been educated in a program to change their behavior; it’s another thing if that person’s new behaviors have an impact on their organization, on their career, or in the world more broadly. …
Behavior change and impact are sometimes done by universities as part of their systematic program review process, whereby they for example monitor the career success and employment rates of their graduates of particular programs. They even do interviews with graduates or their particular employers to follow up and find out how effectively they’re performing in their jobs.
The Kirkpatrick framework is very broadly pertinent. It’s used on the traditional post-secondary process as well as in non-traditional and non-degree/credit program activities.
4. Can the Kirkpatrick model blend non-traditional and traditional conceptions of success?
There are some ways that the two will not ever really meet because there are certain standard metrics that are used on the traditional side, because the program is so standardized that you will never really accomplish for non-credit or non-degree programs. …You can create metrics such as graduate rates and time to graduation that map the progress of conventional students through their program of study, and you can apply those same traditional metrics to the non-traditional post-secondary education that is about offering degree programs, but you can’t apply those same metrics to non-degree/credit programs such as corporate training, or personal interest programming, or even certificate programs often for professional development purposes. There are some things that we will not ever see a perfect match or correspondence between how we evaluate the success of no-traditional programs and how we do so with traditional undergraduate and graduate degrees.
But there are some ways that the two are actually similar. When you’re looking at the measurement of satisfaction… in both contexts you’re interested in the students’ or participants’ feedback on what their impression has been of the quality of their education, the instructor, whether there were clear objectives to the course… those kind of things are virtually universal. Those are the expectations we have of education.
The measurement of learning could also be very similar. Some non-credit instruction isn’t concerned with measuring the learning of the participants. … But in many cases there is an examination of learning, and in those cases the types of assignments, the types of activities used to assess learning are very parallel, in some cases are exactly the same as you would find in undergraduate or degree programming.
The measurement of learning is really no different on the non-degree side than it is on the degree side.
You Might Also Like
