The making of, the Liberals' survey -

The making of, the Liberals’ survey

A Q&A with the CEO and founder of Vox Pop Labs, which produced the government’s divisive survey on democratic values

A voter casts her ballot at a polling station in Quebec City, October 19, 2015. (Mathieu Belanger/Reuters)

A voter casts her ballot at a polling station in Quebec City, October 19, 2015. (Mathieu Belanger/Reuters)

On Monday,—the website where the government is seeking input from the public on electoral reform—went live. Virtually none of the questions touch specifically on voting systems, but rather focus on democratic values and citizens’ broad, theoretical preferences for elections and their government.

The site was built by Vox Pop Labs, a Toronto-based company “specializing in data science and civic technology,” at a cost of $326,500. Maclean’s spoke with Clifton van der Linden, founder and CEO of the company, about how the survey was crafted and what insights it aims to draw from Canadian voters.

Q: How did Vox Pop Labs become involved in

A: We were commissioned by the government of Canada to develop an engagement platform that was focused on democratic values specifically, and to try and reach a broader audience than conventional modes of engagement may be able to reach. My company has done many presentations about our work in digital engagement, and it was on the basis, I think, of our reputation in that space that the government of Canada sought us out to see if this kind of initiative might be possible.

Q: What exactly did your company do in shaping this survey? How do you go about creating something like this?

A: Once we had the parameters of the survey, we worked with an advisory panel of prominent scholars in areas such as research design, survey methodology and electoral politics. We developed a survey that drew from the existing literature on electoral reform in Canada and tried to identify various values that structure that discussion. We then built a very large survey with a number of questions—many more questions that you see included in—and field-tested those questions to random panels of Canadians to ensure that we were controlling for biased survey design or any other deficiencies.

Once we did all that field testing, we then went to quite a large panel of Canadians—more than 3,000—and asked them the remaining questions and used their responses to generate a cluster analysis. A cluster analysis finds correlations in responses to a survey and identifies how different groups of respondents cluster in a specific space.

If you go to and get your initial result, you can see the themes that emerged and how you are positioned on those themes, and also the distribution of Canadian public opinion. Once we had those clusters, which we represent as archetypes in the tool, we were able to look at their positions on issues and develop a narrative around them and also provide a label for them that spoke to some of their positions.

Q: Why did you design the survey the way you did? Why was it designed to turn on broader, almost fuzzier, ideas like democratic values? How was that an effective way to get at attitudes toward electoral reform?

A: That was what we were commissioned to do by the government. That was the mandate that we were given: to focus on democratic values and not on electoral systems. I can’t speak on behalf of the government, but I think the minister [Maryam Monsef] has spoken to the view she takes on this, that democratic values help inform ideas around electoral systems.

Q: So the tool would not, for instance, tell a given voter that proportional representation best fits what they say are their priorities. It doesn’t match up directly with voting systems, is that right?

A: That’s right. A lot of these themes are themes that you’ll also find in the literature on electoral systems. But our explicit mandate here was to focus on democratic values and not electoral systems.

Q: How are those archetypes assigned to people? What’s the cluster of indicators that tells you someone is more likely to fit into one category or another?

A: These archetypes are probabilistic. So they take the aggregation of your responses and make a probabilistic determination as to which archetype you’re most closely aligned with. We try to use that aspect to promote broader engagement, especially engagement online. That initial archetype label and narrative are shareable items that might give people an interesting point of entry into the discussion. Our hope is that users will read that and then continue on through the application to look at some of the more substantive analysis about where they’re situated within these various themes in the discussion on democratic values.

Q: What’s your response to criticism that some of the questions seem designed to elicit a particular answer? There are a lot of questions that use phrasing like “even if” that sound like they’re designed to lean people one way or the other. What’s your response to that kind of criticism?

A: First of all, I would say that a lot of these questions reflect questions in the academic literature. The questions were developed by a team of academics who are specifically trained to develop questions to get at real opinion, and in an objective, dispassionate way. It’s also important to realize that no single one of these questions determines one’s position on a particular theme. The whole point is to take several different items, frame them in different ways and see what kind of variance you get in the response. And once you do that you get a better sense of what a person’s true position is on that issue. We include the “even if” questions specifically to understand the extent of one’s agreement with a particular item.

Q: It sounds like triangulating toward an answer by coming at it from different directions and saying, “Does this still hold for you? How about now?” Is that what you’re saying?

A: Precisely.

Q: Is it true that people can vote more than once, either from different devices or the same one?

A: It’s not really voting.

Q: Responding, then?

A: Yes. People can respond or they can engage with the initiative from more than one device.

Q: How do you prevent stuffed ballot boxes or weigh the results, given that?

A: We have a series of techniques that identify possible entries that are the same user multiple times, or people who are actually trying to skew the results. We have safeguards that are already in place to identify and control for these kinds of multiple entries. In addition to that, once we have the dataset we have a series of screening measures that we apply to ensure that the observations and dataset that are consistent with unique respondents.

Q: Is it possible for you to explain in a way that laypeople will understand how you do that? How you would control for, say, proponents of proportional representation who are highly motivated voting over and over, or other factors that might skew it?

A: Well, there are the common instruments that you would use—things like tracking IP addresses and using cookies and devices that will allow you to detect if a user has used the application before.

Q: If that were the case, would you take the first entry from someone and discount the later ones?

A: Typically, when we’ve identified observations that are assumed to be from the same user, we take the first set of responses. We make a theoretical assumption that the first time they use the application they are really giving an earnest and honest go, and then after that they are likely testing it or revisiting it to see how small adjustments may change their outcome.

Q: Is there anything else you’d like to add?

A: I think we’re very focused on the elements of the initiative, which is great. And we welcome scrutiny of our techniques and our methods. But I think we’re missing out a bit on the bigger picture if we don’t go into the fact that this is in many ways a really innovative way for the government to engage with and consult the public on an important issue. I think that we will reach and engage with Canadians who would never have been engaged through the conventional outreach processes the government usually engages with. Their voices will be reflected in the work that we do in a way that they otherwise wouldn’t be.


The making of, the Liberals’ survey

  1. One thing this survey does prove is that Trudeau is utterly clueless or completely cynical about the how the internet works. Or surveys, for that matter.

    This *survey* is less secure than the National Post firewall but displays more conformational bias than any of their editorials, which are always free.

    This survey will simply confirm the adage “You get what you pay for”.

    Be sure to vote and vote often. Vox Pop Labs and Justin are both counting on it.

  2. You get what you voted for Canada. This country is being run by a group of inept, clueless and utterly useless band of morons lead by a puppet on a string. Don’t blame me I didn’t vote for them.

  3. Actually you can complete the survey repeatedly on the same device – I just did. I think if the Liberals want vague, ambiguous results (“values”) that can easily be interpreted to mean anything this is a well-designed survey. However, if they actually wanted to see how much we wanted them to meet their election promises regarding the election process, a simple survey that explained the alternative processes – first past the post, top choices, etc. and allowed us to choose would have been far more effective, and would have cost us less than $326,500.

  4. As a Liberal donor, party worker, and member, I can say I was ashamed we provided this survey to the Canadian public.

    I guess we know where to go when we want loaded questions and a “push poll” survey! I’ve seen better survey’s in Cosmopolitan magazine, and the only difference was the result wasn’t that “You’re a Leo” – just something equally as inane.

    Complete and utter rubbish, and as a Liberal supporter I can say without hesitation the Minister needs to resign. The opposition are right on this.

  5. The survey said I’m an innovator, for a diverse and inclusive democracy.
    I am the inventor of Binomial STV (against all official elections allowing premature exclusion of candidates); also the Harmonic Mean quota (a purist proportional count).
    The Single Transferable Vote is indeed for diverse and inclusive democracy.
    Google: ERRE>Work>Electoral Reform>Briefs) namely, BC Citizens Assembly on Electoral Reform (September 23).
    Richard Lung.
    Website: Democracy Science; with links to 3 free e-books on election method: Peace-making Power-sharing; Scientific Method of Elections; Science is Ethics as Electics.

  6. Take a few minutes and do the survey. Ha. Ha.

    Whatever you do don’t use the results to justify electoral policy. You’ll look like a real boob.

    I keep getting kicked for my posts. Spread it around ’cause no matter what I post I get hoofed. Ain’t that right Asian/Evangelical darlins’? Some of the darlins’ are real worried about electoral reform. Guess they figure it will be harder to thump a win or something.

  7. With all due respect to the political junky writers for their negative comments regarding the My Democracy on line survey, they do not take into question the level of knowledge and education of the general Canadian population with respect to Canadian politics and systems of voting. If a regulare Canadian were to read a survey written by a political junky/journalist they would not have a clue of what they would be reading therefore would discard it after the first one or two questions. In otherwords I believe the simpler the questions of such a survey would bring more and much better responses.
    R Low

  8. I object to the heading in your story saying it is “divisive” – that is your opinion and not fact. Also the media attempting to form public opinion is becoming very troublesome.

    • ‘media attempting to form public opinion’

      Really? Hmmmm…..I always thought it was the thumpers forming the media’s opinion. Huh.

      News update with my stalkers. I have some new minders. Ain’t that right darlins’? Small town can work for you and against you. Perhaps you wanted to be seen???

      Buzzzzzz, buzzzz….You love changing the frequency don’t you?? Follow the signal eh?

    • The media is reporting on a problematic issue…that being the fact that Trudeau promised electoral reform based on 39.6 percent of the popular vote. Then he formed a committee and the committee talked to Canadians and experts and when it didn’t get the results Trudeau wanted, he balked and put out this survey. He is suffering from a severe case of confirmation bias. He is trying as hard as he can to make the results of his enquiries fit what he has decided is best for Canada. If they don’t, he will ditch the effort. He doesn’t jump at the referendum that Canadians want because he can’t control the outcome. Wake up and smell the coffee. It isn’t complex. It is simple politics. Thank goodness for a free press that keeps apprised of what is transpiring in this country.

  9. The survey cost $326,500? Did that include the cost of producing and mailing postcards to 15 million households and all the social media advertising? The entire ERRE budget which included hosting 200 witnesses and traveling the country holding town halls was only 300K… Imagine the kind of public engagement the committee could have generated with a doubled budget!

  10. I as a Canadian am very upset that we are being controlled by 1/3 of the French population! We as a nation have to let” THEM GO” where ever does not matter.The amount of money we spend on keeping them is out of this world! I am from New Brunswick and i see the full force of the French as it takes over all aspects of our lives ,The rest of the country is not in this position,so to be FRENCH is to be taken as a blessing? I as a Canadian feel left out ,because of the treatment that the FRENCH receive.

  11. 4142 478196Dead pent subject matter, thanks for entropy. 788355