CLICK HERE for information on the results of three major national student surveys: National Survey of Student Engagement (NSSE), Canadian Undergraduate Survey Consortium(CUSC) and the Maclean's University Student Survey

In 2004, the Ontario government commissioned former premier Bob Rae to prepare a report on post-secondary education. Raising concerns about the quality of undergraduate education, Rae called on Canada’s largest province to establish benchmark data on “key aspects of higher education,” and for “evaluating and publicly reporting on quality and system performance.” To this end he recommended that all Ontario universities participate every two years in a long-standing American student survey: the National Survey of Student Engagement, or NSSE. “I am a great believer that if you can’t measure you really can’t make change happen,” Rae recently told Maclean’s.

One way of measuring universities is to ask the opinion of their customers: the students. Are they satisfied with their education? Their professors? Their choice of university? To try to answer those questions, we present the results of three national student surveys: the NSSE, a Canadian survey known as the Canadian Undergraduate Survey Consortium(CUSC), and a survey conducted by Maclean’s to cover off the small number of universities that took part in neither NSSE nor CUSC. Most of these results have never before been made public.

More than 70,000 Canadian students participated in these three national surveys, across nearly every university campus.

NSSE and CUSC each ask dozens of questions about specific aspects of the undergraduate experience—inside the classroom and beyond. Both surveys were commissioned by the institutions themselves. NSSE, an American survey that 28 major Canadian universities took part in in 2006, focuses on student engagement. Most of the questions asked on NSSE are an attempt to find out how students are spending their time and how “engaged” they are with their schools, their professors and their peers. It is mostly about asking students what they did, not how they felt about it. NSSE does include a few satisfaction questions, however, which you will see featured on pages 31 and 32. The NSSE surveyed undergraduate students in first year and fourth year.

The CUSC survey also looks at detailed aspects of the undergraduate student experience, but takes a slightly different approach. Unlike NSSE, it includes many questions asking students to assess how satisfied they are, and where they would like to see improvements. The 2006 CUSC survey was conducted amongst a sample of 1,000 graduating-year students at participating universities.

There are nine Canadian universities that did not take part in the 2006 CUSC or NSSE surveys. To provide readers with feedback from their students, last January Maclean’s asked those universities to invite their students to take part in a web-based survey based on the CUSC, using CUSC methodology and CUSC questions. Eight of the nine universities agreed.

Of the 47 universities appearing in the annual Maclean’s ranking of universities each fall, only Université de Moncton did not take part in any of the three surveys. It is not listed in any of the charts. The other university that is missing is York. It took part in both NSSE and CUSC, but the university has so far declined to make public its results.
So what do the surveys say?

Overall, students at smaller, undergraduate-focused universities say that they are generally more satisfied than students at larger, research-oriented universities. There are exceptions to this trend, with larger research powerhouses such as Queen’s, Guelph, Western and Waterloo getting high marks from their students. But on the whole, small schools tend to do much better than larger institutions.
When the CUSC survey asked students, “has your experience at this university exceeded, met or fallen short of your expectations?” a substantial majority at all universities said that their expectations had been exceeded or met. However, at a surprising number of universities—all larger universities such as Calgary, Simon Fraser, Ottawa, Montréal, UBC, Dalhousie and the University of Toronto Scarborough—around a quarter of graduating-year students say that their university experience had fallen short of their expectations.
Similarly, on the NSSE, in which both first- and final-year students took part, two broad satisfaction questions also elicited high overall positive responses, and not just at smaller undergraduate universities. When asked to “evaluate their entire educational experience,” a majority of students answered either “excellent” or “good.” However, while more than a third of students at many universities were willing to describe their educational experience as excellent, fewer than one in five senior-year students at Dalhousie, Lakehead, Laurentian, Laval, UBC and Ottawa were willing to give their education top marks.

When asked, “if you could start over would you go to the same university?” the majority of students at all universities answered “definitely yes” or “probably yes.” Once again, the trend favoured smaller universities, but some larger universities also did well. For example, fourth-year students at a number of universities with a wide range of master’s and doctoral programs gave their schools grades above the NSSE average.

But by the time they reach fourth year, fewer students at most universities were willing to say that, “definitely yes,” they would choose the same university. For example, Ottawa’s score went down from 35 per cent to 17 per cent. Even top performers Queen’s and Western both declined between first year and fourth year from 60 per cent to 45 per cent. It seems that students are, for whatever reason, generally less likely to recommend their school after completing their course of studies there. Notable exceptions are Brock, UPEI and Trent.

When asked on the CUSC survey about the quality of teaching at their university, at no institution did a majority of students choose “agree strongly.” Students at smaller campuses were once again most likely to indicate the highest level of satisfaction. And students at a number of other larger, arguably more prestigious universities—such as UBC, Saskatchewan, Calgary, Ottawa, Simon Fraser and U of T Scarborough—were least likely to agree strongly that they were satisfied with the quality of their university’s teaching.
Though university students clearly have complaints about specific aspects of their learning experience, most report they are “satisfied” or even “very satisfied” with their university experience. On the CUSC survey, 89 per cent of students indicated that they were “satisfied” or “very satisfied” with their “overall educational experience,” while 82 per cent said their university had “met” or “exceeded” their expectations. Another 89 per cent also said they were “satisfied” or “very satisfied” with their “decision to attend their university.” In the Maclean’s survey, more than 90 per cent of students said they were “satisfied” or “very satisfied” with the “overall quality of education” and with “their decision to attend their university.”

Ontario schools taking part in the NSSE also presented their students with a few Ontario-only questions. When asked which areas inside the classroom were most in need of improvement, Ontario first-year students chose “improving the quality of teaching assistants.” Areas outside the classroom they said were most in need of improvement were “increasing contact with professors” and “working toward a better social environment.” Areas most cited by upper-year students as needing improvement were “the quality of course instruction by professors” and “providing students more opportunities to undertake research with faculty.”

So what do all of these results mean?

One university whose students give it among the lowest satisfaction ratings on the CUSC survey is the University of Manitoba. Across town, students at the smaller, undergradute-focused University of Winnipeg gave their school among the highest satisfaction ratings. Tessa Vanderhart, a third-year political science student at Manitoba, calls these results “not unexpected. I think student satisfaction is higher at Winnipeg because everyone knows each other there.” As to the question of the validity of the results, “I think the perceptions people have [of the two universities] are true, so people generally know what they are getting themselves into.”

Is a university with higher student satisfaction a better university? “I think one should take student survey responses seriously,” says Carl Wieman, Nobel Prize winner in physics and director of the Carl Wieman Science Education Initiative at UBC. “But one should not automatically assume that the best policies are to follow everything students say they would prefer.”(You can read the full interview with Wieman on page 12, or listen on the web at macleans.ca/university.)

For example, argues Wieman, the finding that students at smaller universities tend to have higher satisfaction levels than those at larger universities may not tell us anything about the quality of education. “I know everybody likes something sort of more personalized,” says Wieman. “And that does happen better at small places, and so people are happier. Whether the actual education they get is better is quite a different question, and frankly that’s not something students are necessarily in a position to evaluate, at least while they’re going to school.”

For NSSE participants, the key results are not the answers to the satisfaction questions, but the school’s performance on five performance benchmarks, measuring “engagement.” NSSE assumes that engagement is a correlate of quality, or a measure that indicates that more and better learning is likely to be taking place.

Each university participating in the NSSE receives a benchmark report comparing scores from first- and fourth-year students on key questions with those of other participating institutions, including all of their Canadian and American peers. The key questions are then grouped together in five broad benchmark categories, each with an overall benchmark score: level of academic challenge faced by students; amount of active and collaborative learning; quality of student-faculty interaction; availability of enriching educational experiences; and supportiveness of the campus environment.

So what do the benchmarks tell us about the undergraduate learning experience at Canadian universities? For starters, Canadian universities at both the first- and final- year level compare quite well to their American peers in the benchmark areas “level of academic challenge” and “supportive campus environment.” The academic challenge measure is made up of scores on questions in such areas as “number of assigned textbooks,” “number of written papers,” and “coursework that emphasizes analysis of the basic elements of an idea.”  

However, all Canadian universities participating in the NSSE scored below the NSSE benchmark average on the “student-faculty interaction” benchmark(see table, page 36), and were also behind their American peers on “active and collaborative learning”(see macleans.ca/university for the benchmark figures not presented here).

Why? In 2004, a handful of Canadian universities participated in NSSE for the first time, and showed the same low scores. It may come down to nothing more than resources: Canadian universities have fewer. “I was surprised,” says Chris Conway, director of institutional research and planning at Queen’s University. Given the resource disparity, he expected his institution and others to be behind their American peers, but not to that extent. “That was our first hard empirical evidence that showed resourcing matters on those apsects of learning that are directly related to student-faculty interaction.” Canadian universities do poorly on these benchmarks, relative to their American peers, because the Canadian institutions, with smaller per student budgets, have fewer professors for each student.
A growing number of Canadian universities are taking part in NSSE, and Bob Rae is not the only one pushing a greater reliance on its findings. “We want to be really good and give the best that we can to our students and that’s what drives us,” says Harvey Weingarten, president of the University of Calgary. “It drives our behaviour, it drives our resource allocation, it’s why we do things like NSSE; that’s why we listen to what NSSE has to tell us.”

Many universities prefer NSSE to CUSC, and are less comfortable with the kind of student satisfaction questions that CUSC asks. Despite Queen’s high standing on satisfaction surveys, Conway maintains that satisfaction scores aren’t always useful, because they are highly dependent on the expectations that students have going in to a university. As a result, he says, questions of satisfaction may not be comparable among universities: “If high expectations are more or less met, then students respond accordingly. Or if they have low expectations and nothing happens to subvert that, then students also give relatively consistent satisfaction scores.” Conway doesn’t reject satisfaction questions, and admits that they are “roughly correlated, but not particularly well, with NSSE engagement scores.” But he doesn’t view satisfaction scores as being as useful as the more “empirical” measures of NSSE. “Satisfaction readings have to be taken with a grain of salt,” says Conway.

For example, a student at a primarily residential university, with all the social life and student interaction that implies, might be more satisfied than someone at a commuter school. And, given that they attend a small, residential school, NSSE results might also indicate that they are more engaged. Conway cautions against making too much of student satisfaction data. “On average, Canadian students are reasonably satisfied and that’s a good story. I wouldn’t make hay out of what appear to be minor differences."

To help improve the student experience, and in response to the findings of these surveys, universities are developing innovative programs and support services to help improve the student experience. In 2005, the University of Calgary established the Quality Money fund. The student government, in consultation with the student community, can direct money to projects that improve the student learning experience. Last year, the student union directed $1.37 million to establish a variety of projects such as an undergrad research program in health and wellness, a class-size reduction program, and the establishment of a Teaching Excellence Awards program. UBC hired Nobel laureate Wieman to study and reform the teaching of science at the undergraduate level. And many universities are introducing new ways to deliver first-year programs that give students a chance to experience small seminar-type settings that are more often associated with upper-year courses. These small learning groups are particularly important at large universities where first-year students often sit in large lecture halls with hundreds of others, rarely getting a chance to ask a question or discuss ideas with the professor or fellow students. In fact, 17 per cent of Canadian first-year students told the NSSE that they have never asked a question in class, compared to just three per cent of their American peers. All Canadian universities are trying to address this problem. For example, the University of Toronto recently started First Year Learning Communities(FYLC)in the faculty of arts and science, bringing together groups of 24 students in the same sections of first-year courses in a regularly scheduled meeting facilitated by an upper-year peer mentor. The meetings include social, developmental and academic programming. In addition, a staff and faculty adviser attend the meetings.
Even though universities rely heavily on surveys to improve the quality of the undergraduate learning experience, some universities still refuse to make this information public. Where universities declined to provide this data, Maclean’s filed access requests through provincial Freedom of Information and Protection of Privacy legislation. Several universities released data as a result of the filings. As we went to press, only York’s CUSC and NSSE results were outstanding. That request is still winding its way through the freedom of information system. We will be publishing these results when they become available.

WEB EXCLUSIVE DATA: | Active and collaborative learning |

WEB EXCLUSIVE DATA: | Enriching educational experience |

WEB EXCLUSIVE DATA: | Supportive campus environment |

WEB EXCLUSIVE DATA: | Non-academic learning |

Print Article    Send to a Friend    Write a letter to the editor

Digg this Stumble It!   Post to del.icio.us Seed Newsvine   Share on Facebook Technorati links