What’s in a performance indicator?

Whether or not the public is getting good “value” for money invested in higher education is a question that naturally creates tension between universities and governments. Since the late 1990s all Ontario universities have been mandated to make public certain “key performance indicators.” A small, in fact very small (two per cent) of public funding for the province’s universities is tied to performance, with low performing institutions seeing very little, if any, of that money.

Last week, the Ottawa Citizen devoted a report to one such indicator, graduation rates, and how Ottawa’s schools “stack up.” Although graduation rates, like other key performance indicators provide a seemingly objective measure of “value” or “quality” or “accountability,” their usefulness should not be overstated.

As the Citizen reported, there is a significant variation in graduation rates among Ontario institutions. At the bottom of the list is Carleton at 63.7 per cent, while at the top sits Queen’s at 86.9 per cent. The provincial average is 75 per cent, which is higher than the national average.

Graduation rates tell us whether students graduate from the same institution where they began their studies. They don’t tell us whether students who leave earn a degree elsewhere. But, universities that are less likely to lose students to other institutions, are often less likely to lose students to attrition (dropping out).

For example, in addition to Queen’s relatively high graduation rate, it retains over 90 per cent of students from their first to second year. Only a very small percentage of students still attending Queen’s in their second year will fail to earn a degree.

The obvious, and seemingly logical, conclusion to draw from these numbers is that universities with higher graduation rates provide a superior quality of education. This thesis was popularized by Milton Friedman in the 1980s, when in his classic Free to Choose he pointed to enormous variations in dropout rates between American public (and thus tuition regulated) universities, and elite private colleges that can set their own (often very high) tuition levels.

Friedman found that at some public institutions as many as 45 per cent of students who began their studies failed to earn a degree, compared to private institutions where the attrition rate was as low as five per cent. A 1991 study published by the Association of Universities and Colleges of Canada came to a similar conclusion, finding that attrition rates among Canadian public universities was, on average, 42 per cent. The Canadian Council on Learning speculated a similar number in a report last December.

If taken at face value, Friedman’s analysis provides supporting evidence for the complete deregulation of tuition. Giving universities greater control of how much money flows into their coffers will naturally lead to a higher quality of education, with one of the apparent outcomes being lower attrition rates.

But, such assumptions fail to recognize the role of the student in earning an education, as opposed to a university in providing one. Elite universities are not only more discriminating in who they let in their doors, their reputation for providing high quality education (real or perceived) attracts elite students, whereas non-elite universities can attract virtually everyone.

High tuition might be linked, in fact it probably is linked, to the level of quality, but whether the level of quality is linked to student success, or whether the number of students graduating from a particular university is linked to quality are different questions.

A much-cited 1999 study by American economists Alan Krueger and Stacy Dale found that the quality of a university matters less than the quality of the student. Their survey of 519 undergraduate students from the 1970s indicated that those who were accepted to a highly selective university but attended a less selective school instead did just as well in terms of future earnings as students who actually went to the highly selective university.

Motivated and capable students can succeed in pretty much any institution. The corollary is that unmotivated and incapable students could be likely to dropout no matter where they went.

A Statistics Canada report last year found that students who dropout share similar characteristics to students who do not attend university, such as not having parents who have a degree, or not performing well in high school. Student dropouts are also more likely to neglect their studies from day one, or complain that university is just not for them.

It might just be that all that a high graduation rate can tell us is that a university is attracting a greater percentage of high quality, low risk, students. This raises a prominent objection to performance based funding: that it provides incentives for universities to alter their operations to satisfy the indicator, such as raising entrance requirements in an attempt to maximize the percentage of high quality students in attendance, without having to improve its ability to attract such students.

Graduation rates may tell us about student choices, which does not necessarily indicate the level of quality. Or they may tell us how attentive a university is to students who just might not be suited to a higher education. Or, far from indicating a high quality institution, a high graduation rate could simply be suggestive of low standards. Then again, it may tell us nothing at all.