There is a notion that higher education—encompassing everything from the institutions that deliver it to the ranking systems that judge it—should be more student-centric. But what does student centricity actually mean? In this interview, Robert Kelchen tackles that question, sharing his thoughts on what a student-centric ranking system would look like and reflecting on the most significant challenges standing in the way of the adoption of such a system.
The EvoLLLution (Evo): Why is it important to have a ranking system that provides students with information on different college options?
Robert Kelchen (RK): Gathering information on different college options can be a time-consuming process, particularly for students who are the first person in their family to attend college. College guidebooks and rankings can provide a good starting point for students as they work to develop an initial list of colleges to consider. However, most of these sources focus on traditional college students—current high school students who are planning to attend a four-year, residential college on a full-time basis and have at least some level of financial support from their parents.
Higher education needs to stop thinking just about the stereotypical 18-year-old when developing policies and practices, and rankings providers are no different. Nationwide, just over half of all students are both attending full-time and between the ages of 18 and 24. Adult learners represent the majority of students at for-profit colleges, as well as nearly 40% of students at community colleges. These students are less willing to travel to attend college due to family and work obligations, often constraining their options to a handful of open-access local institutions and colleges with online programs.
Therefore, the rankings systems that work well for traditional-age students who are more geographically mobile and have higher abilities to pay for college will not work as well for adult students, who are rapidly becoming the new majority in American higher education. Ideally, there would be a separate ranking system focusing on the needs of adult students and recognizing that they often have different needs and goals than 18-year-olds.
Evo: What are some of the limitations of the more popular ranking systems currently in use?
RK: As the methodologist for Washington Monthly magazine’s annual college rankings since 2012, I’ve been able to see both the benefits and limitations of college rankings for students. While college rankings provide concise information about key performance metrics to prospective students and the general public, this information is only able to reflect a broad picture of a college’s performance. Information on the outcomes of students in particular programs is rarely available in any rankings, and precious little information about employment or earning outcomes was available before the U.S. Department of Education released the updated College Scorecard last fall.
College rankings are also limited to items that can be easily measured. U.S. News has the financial resources to join with a few other groups to commission a survey of four-year colleges to collect some additional information, but I’m the only person to work on the Washington Monthly rankings—and I’ve got a demanding day job as a tenure-track assistant professor. As much as I would love to include measures of student learning, civic engagement, or job satisfaction into rankings, collecting that type of data is virtually impossible.
I’m also concerned about college rankings, such as U.S. News, that reward colleges just for being more selective and spending more money. While there is a correlation between both selectivity and resources and higher graduation rates, it is far from clear whether these highly ranked colleges are adding value to already top-notch students or just watching over them for four or five years before they are ready to graduate and join the workforce. Given that most students are attending modestly or minimally selective college, I feel it’s more important to focus rankings on colleges that do a good job educating the students they have instead of the students they might like to have.
Evo: To your mind, how would a truly student-centric ranking system differ from the options currently available?
RK: A truly student-centric ranking system would give students the opportunity to create personalized rankings based on their own characteristics, intended program of study, and willingness to travel to attend college (or willingness to attend online). In part due to data limitations (more on this later) and in part due to the difficulty in developing a feasible personalization system, most rankings today are one-size-fits-all—even though students want to know how students like them have fared at a particular college.
However, there are a few things that can be done with existing data and technology to help customize rankings. Rankings could highlight demographic information that could be useful to prospective students. For example, the Washington Monthly rankings highlight the percentage of students receiving federal Pell Grants as an indicator of economic diversity. Prospective students from low-income families may be interested in this metric to see whether students like them attend that college. Some outcome measures are also available for certain subgroups of students. For example, graduation rates are available by race/ethnicity and gender, while loan repayment rates and earnings are available by first-generation status.
Students could also use an online tool to choose which metrics are most important to them in a rankings system. Don’t care about how well faculty members are compensated (7% of the U.S. News ranking in 2015)? Cross it out. Care more about student-faculty ratio (1% of the ranking)? Give it more weight. Allowing students to play around with the weights—and to substitute in subgroup-specific outcomes for overall outcomes when available—would help to personalize rankings, but at a sizable cost to the college rankings providers to improve their technological capacity.
Evo: What are some of the challenges standing in the way of providing more drill-down, even program-level data for students to peruse?
RK: The only way systemic, detailed program-level data will get into the hands of students and their families is if the federal government makes it available. Some states (such as Florida, Texas, and Virginia) already make this type of information available, but students living in or considering colleges in other states don’t have access to detailed outcome information. The U.S. Department of Education is working to collect some program-level data through its College Scorecard tool, but it will be several more years before that additional information is available to the public.
At many colleges, program-level or subgroup-level data would run into issues of small cell sizes—meaning that a college may only graduate three or four philosophy majors per year. Currently, the Department of Education typically avoids reporting earnings data for cases with fewer than 30 students, which would include most majors at small colleges and even quite a few majors at larger colleges. To get around this, outcomes could be released for general areas of study (such as STEM or the social sciences), but this doesn’t give as detailed of information as some students might like or help students who decide to change their major at some point in college.
The best way to get detailed data on student outcomes for particular types of students is to use student-level data instead of data aggregated to the program level. Although top-notch data systems at the state level use student-level data, the federal government is unable to do so due to a 2008 ban on so-called “unit record” datasets. Although a switch to unit record data could actually reduce the reporting burden on colleges, and bipartisan legislation has been repeatedly introduced in Congress to make this change, no changes to the law appear imminent.
You Might Also Like
