TORONTO – Want to be a guinea pig for Netflix, get early access to new features, and help shape how the video streaming service grows and evolves?
If you’re a relatively new subscriber, you may already be a beta tester. Or perhaps you used to be.
At any given time Netflix is conducting hundreds of tests across its website and other platforms. Some are based on bold new ideas and are immediately obvious to its test subjects, while others are being run invisibly beneath the user interface.
Users who have been with Netflix since the company launched in Canada in September 2010 aren’t likely to be in on the testing, but newer members almost certainly are, whether they realize it or not, said Netflix’s chief product officer Neil Hope.
“One of our limited resources is users to try out new ideas, so I would say it’s the unusual individual who doesn’t end up in some kind of test at some point,” said Hope.
“Most testing is done with brand new users who don’t have learned behaviours to unlearn, that gives us the cleanest read, and so the typical new user coming in will end up in half a dozen or a dozen different test experiences, most of which will be very trivial and minor and invisible.”
A year’s worth of testing culminated in the recent release of Netflix’s Profiles feature, which allows users to create multiple log-ins within the same account. And it took almost two years of testing before Netflix released My List, which allows users to bookmark titles to watch later. Netflix relies on the statistics generated by user tests to inform its decisions, Hunt said, rather than following the so-called “HIPPO” philosophy. That acronym stands for “highest paid person’s opinion” and refers to an organization’s preference to make top-down decisions.
“That seems like a really bad idea and so the nice thing about the strategy that Netflix uses is, the customer speaks, the customer’s behaviour and actions on the service tell us what the winning proposition is based on how it affects the metrics we care about,” he said.
“We like to test lots of different stuff, as simple as the size of an image, or the font on the page here or there, or a colour or a shape, all the way up to large and complex features like Profiles and queues.
“This is our whole innovation strategy, try out lots of ideas, quickly get them up in front of customers, measure and see what works well, feel that result, and then move on to build new and different ideas.”
But there’s a fine line between trying out lots of promising ideas and pushing out countless tests just for the sake of experimentation, he added.
“We’ve become very sensitive to the notion that things winking in and out and changing randomly is a bad, bad idea and so we generally expose people to testing ideas for the long term,” Hunt explained.
“So a test assignment happens when a new member joins up, the test assignment will stay sticky to that account for the long term, for months, so a particular person is only ever going to see one version or one variation (of an idea). And until we’ve decided that we’ve got a winner and it’s time to roll it out more broadly, we won’t be changing anyone’s existing behaviour.”
The good news for Canadians who are accustomed to big tech companies launching new products here months or even years after the U.S. is that Netflix has decided to release its new features to all its markets simultaneously.
One exception is a feature called Max, which is currently being tested only in the U.S. and only on PS3 video game consoles.
It’s a major departure for Netflix, offering an interactive way for users to select something to watch, as opposed to having them simply click up, down, left or right through a static interface. Gamers who remember the “You Don’t Know Jack” trivia games will recognize the shtick of Netflix Max, which guides users through the process of selecting a movie or TV show by challenging them to play some mini-games or offering a random mystery selection.
“The whole Max talking user interface is a bit of a prototype for a whole new type of user interface,” Hunt said.
“We’re interested in continuing to push the envelope of what’s possible and what’s interesting and learn and understand and it’s an experiment with our users too. Users have to figure out if this is a good thing and if they want to engage with it and it’s sort of a mutual learning process.
“Eventually maybe we end up in a place that’s quite different from where we are today.”