The following interview is with Leah Lang, manager of the Core Data Service at EDUCAUSE. At the recent EDUCAUSE conference, Lang presented on methods institutional IT units can use to align themselves with institutional strategic objectives, and further discussed how IT units can measure their success. In this interview, Lang expands on those topics and shares her thoughts on some of the metrics IT leaders can use to measure their operational efficiency.
1. What are some of the most significant challenges postsecondary IT administrators have when trying to quantify the efficiency of their operations?
There are really three main challenges. One is knowing what data to use; another is getting the right data and, finally, interpreting and knowing what to do with the results.
In terms of knowing what data to use, there’s a lot of data around, so if an institution is trying to demonstrate value, they may be wondering if it’s more important to highlight the number of tickets the helpdesk is answering or number of problems solved. If they are trying to demonstrate volume, they might be wondering which data tells a more unbiased story; is it number of tickets per FTE (full-time equivalent) or is it just number of tickets? There are many different types of data points and there are many different ways to slice the data. You have to carefully consider which data tells the story you’re trying to tell.
Sometimes, if you’re collecting certain metrics about efficiency, staff are concerned that it reflects poorly on their performance. IT leaders must communicate the reason for collecting the data and cultivate an environment in which staff are willing and motivated to provide the best data.
In terms of interpreting and knowing what to do with the results, a CIO or institutional leader—or anyone using metrics—might be wondering, “So I closed more tickets this month than last month” or, “I know that this month I closed more tickets than ever before, what now, what do I do?” The best thing to do with data is not to act but to ask questions. Metrics are just indictors of what’s going on around you and they are most helpful in identifying anomalous circumstances. When anomalies occur, good or bad, it’s important to determine the cause and then act on the results of that investigation, not on the data directly.
2. At the recent EDUCAUSE conference in Orlando, you presented on a new generation of IT assessment tools aimed at helping institutions understand the relative efficiency and productivity of their IT services. How do these tools help IT leaders overcome their challenges?
EDUCAUSE collects the data that IT leaders need to track efficiency at a high level. This takes problems one and two that I just mentioned off the table for many strategic issues IT leaders face. We’re making it easier by taking the guesswork out of which data to collect and we also teach institutions how to get those data points so that the right data is recorded. We instruct anyone answering our surveys on how to get that data within their organization and best practices that other institutions use to get at that data. We provide specific definitions about which metrics we want them to gather. We really train the survey respondents in providing the proper data so that our database is filled with accurate data that provides an apples-to-apples comparison between institutions.
That data we collect through the EDUCAUSE Core Data Service, in addition to new tools we’re developing, will take the guesswork out of the third hurdle I mentioned: Interpreting and knowing what to do with the results. We’re trying to not only collect the important metrics and report them back and make sure that they’re consistent between institutions, but we’re also trying to add a layer of analysis on top of that so that those using the data won’t have to guess about what the results are telling them or what to do with the results.
3. What are some of the metrics you think are central to the measurement of efficiency in the postsecondary IT context?
EDUCAUSE has been collecting this data since 2002 and over that time we’ve identified a set of core metrics that CIOs use most to evaluate IT at their institution. These consist of data points like central IT expenditures per institutional FTE, central IT expenditures as a percentage of institutional expenditures, IT staff as a percentage of institutional FTE and the inverse metrics—IT FTEs per institutional FTE.
These can all help to normalize an assessment of IT at your institution so you can more easily compare to other institutions. CIOs use this information to determine if they’re maybe over- or under-resourced, or if their portfolio of services is in line with their peers.
4. Is staff-student ratio alone a reasonable way for institutions to determine their levels of efficiency?
It really depends on what the question is. If the question is “What is the most efficient way to deliver the higher ed experience to the most students” then it might be appropriate to use the staff-student ratio to determine if you’re hitting that sweet spot between too many staff and too few. If the question is “What is the most efficient way to produce successful graduates,” there are probably other better metrics that should be considered as well, such as cost per student perhaps.
If you are trying to assess any type of situation or use metrics to determine how well you’re meeting your goal, don’t just use one metric. Use more than one metric because you want to really see all sides of the situation. For example, with the EDUCAUSE data we collect, we’ve noticed that some institutions have really high central IT expenditures as a percent of institutional expenditures. The amount of money they’re spending on IT is really high compared to institutional budget, but when you look at how much money they’re spending per FTE—maybe because they have more students—[the figure is much lower.] If you were only to use one metric in that situation, you may come to a different conclusion than seeing both metrics.
Completing our annual Core Data Service survey gives IT professionals access to hundreds of free performance metrics and this year the survey is due by November 7. Anyone interested in completing the survey can go to www.educause.edu/coredata. It’s free to any institution and you don’t need to be an EDUCAUSE member to participate.
This interview has been edited for length.
You Might Also Like
