Case Study#3 : Inside Booking.com's experimentation culture
Case Study#3 : Inside Booking.com’s experimentation culture ATTACH
It is estimated that any given point in time, 1000 simultaneous tests are running on Booking.com and it could go as high as 30,000 tests in a year.
Stuart Frisby, Booking’s Ex-Director of Design, had set a few guidelines in order to promote and sustain experimentation culture:
- No HIPPOs (highest paid person’s opinions)
- Every decision is a democracy, but test every decision
- Trust your tools
So, do all experiments succeed?
No, in-fact majority don’t. It is estimated that only 10% of experiments run at Booking.com succeed.
How are experiments setup so easily?
Every 3rd employee out of 4 (of all product and tech employees) actively use the experimentation tool that Booking.com has built in-house. The tool has standard format and templates that allow the employee to setup an experiment in <2 mins if all metrics are known and development needed for setup experimentation has already been done.
A/B Test Back in 2020, Aleksander & Benjamin (Microsoft), Pavel (Outreach) and Lucas Vermeer (Booking.com) developed a A/B testing flywheel - a circular framework that means companies need to keep investing iteratively into experimentation in order for it to become company culture. The five steps in this flywheel are as below and they are super important for any organisation looking to bring experimentation into their culture:
- Measure value to decision making: Running more A/B tests is important to generate more value related to decision making through experimentation.
- Increasing interest in A/B testing: As teams run A/B tests, they should also share their learnings with others - backed by training and support. This motivates other teams to try as well.
- Investing in A/B testing infrastructure and data quality. As A/B tests start generating value, more resources can be justifiably allocated to make an A/B testing program successful.
- Lowering human cost of A/B testing. Eventually, the goal is to ensure no one has to spend a lot of time to start an experiment. Hence, it is important to lower the time and effort involved for each A/B testing to start.
Power of Compounding（复利） ATTACH
Assuming 10% success rate of experiments and 1% uplift in revenue per test (Source), you can see below how revenue uplift can be achieved and how it starts compounding (just like stock market returns) as you run more experiments:
Learnings from Booking.com
Building experimentation culture (where there is none) is tough. Through this post, I wanted to share how experimentation can be setup slowly (the flywheel), how it can impact revenue (analysis above) and how Booking.com has benefited from it. Here are key takeaways:
- The goal is to create an experimentation culture , and not just buy experimentation tools and force teams to do it.
- Doing many experiments and failing is better than doing less experiments given the 10% success rate.
- Experimentation culture is built step by step by sharing impact and encouraging other teams to do it. It is more of cross-pollination rather than just standardising it across teams.
- Most importantly, Booking.com democratises decision making and every decision becomes data driven. This is apparent from below anecdote:
“When Booking’s previous CEO first arrived from the US, he presented a redesigned logo to the staff. People said “that’s great; we’ll check it with an experiment.” He was baffled but had no choice. The experiment would determine if the logo could stay.”