Netflix: Lessons in Experimentation
Netflix: Lessons in Experimentation
On the other hand, launching anything as an experiment takes more analytics and engineering work, and typically also slows down forward progress. If you already have a vision of the future, does it make sense to test your way there in twice the time?
For Netflix, the answer has been ‘yes’ for nearly its entire quarter-century history.
How does Netflix manage all the tradeoffs of experiments?
The Story
1997
Lesson 1: Experimentation Works in the Real World as Well as Software
Since the earliest days of Netflix, before it even began as a DVD mailing company, it has had a culture of experimentation.
Reed Hastings had sold his company Pure Software for $700M and was CEO of the new combined entity Pure Atria. with Marc Bernays Randolph began discussing the opportunity to mail movies.
Reed saw the DVD format proliferating in Japan but had never used one himself. So he drove to the store and got himself one. He then mailed it. To his great surprise, it came back in great shape. It worked.
1998
Marc and the team went about testing out different types of packaging. Many of the employees’ families helped test…They just shipped them back to test disk durability in the mail. Netflix: How a DVD rental company changed the way we spend our free time
After various iterations, the team landed on Netflix’s now-famous padded paper sleeve. It was cheap and durable enough to give the team the confidence to launch.
Netflix made its official debut on April 14th, 1998…does not mean users will come.Netflix would experiment with hooks like free trial to nab the growing market of DVD watchers.
1999
Lesson 2: Experimentation Is Great for Business Models
in 1999 was experimenting with a subscription model.
the team built out subscriptions for a small alpha group. The numbers were promising. As a result, they made the change for everyone.
2000 Movie lovers everywhere were subscribing to the service.
Lesson 3: Personalization Is a Great Canvas for Experimentation
Early implementations of the Netflix system used IMDb and user rating data to create a content-based predictor. Personalization would take the content-based methods that had driven Netflix’s early algorithms and extend them to personal attributes.
People were watching more movies when subscribed to Netflix, getting more joy, and therefore recommending to friends. Netflix revenue more than 7x’d to $36M.
2001 Revenue 2x’d from $36M in 2000 to $76M in 2001.
2002 On January 4th of 2002, the Netflix S-1 dropped.
Whatever the valuation, the team would continue to work on the core business throughout this time. And the core business was Personalization.
2003 2003 was the year Marc Randolph left Netflix. Reed Hastings became sole CEO.
Netflix would more than 2.5x its subscriber base from 1M to 2.6M over the course of the year.
2004 This theme of personalization as Netflix’s investor positioning would continue in 2004.
2005-2006 As Netflix began to reach scale, growth slowed. Instead of 2-3x, subscriber growth hit 51% and 19%.
Part of this was a strategy. Netflix was preparing for the next leg of its business: Streaming.
2007 Subscriber growth was a minuscule 19% for the year. Netflix was offering a service for high-bandwidth consumers before they even had high bandwidth.
2008 Eventually, users got the bandwidth, and habits changed.
Lesson 4: People Outside Can Help
2009 Reed and the team were ready to drive a broader experimentation agenda with personalization to reduce content costs.
Their solution was the $1M Netflix Prize. Released in 2006, no team was able to crack it for three years.
in 2009, the winning teams ended up submitting an algorithm that performed 10.05% better than the one Netflix used.
2010