Building a Culture of Experimentation
Building a Culture of Experimentation ATTACH
In December 2017, just before the busy holiday travel season, Booking.com’s director of design proposed a radical experiment: testing an entirely new layout for the company’s home page.
Booking.com’s CEO at the time, was skeptical. She worried that the change would cause confusion among the company’s loyal customers.
Anyone at the company can test anything—without management’s permission.
They’ve helped Microsoft’s Bing unit, for instance, make dozens of monthly improvements, which collectively have boosted revenue per search by 10% to 25% a year. (See “The Surprising Power of Online Experiments,” HBR, September–October 2017.)
If testing is so valuable, why don’t companies do it more? After examining this question for several years, I can tell you that the central reason is culture.
Cultivate Curiosity
It’s actually less risky to run a large number of experiments than a small number.
At many companies the risk associated with experiments makes managers reluctant to allocate resources to them.
Many organizations are also too conservative about the nature and amount of experimentation.
Insist That Data Trump Opinions
Note that I’m not saying that all management decisions can or should be based on online experiments. Some things are very hard, if not impossible, to conduct tests on—for example, strategic calls on whether to acquire a company.
Democratize Experimentation
And everyone can see the real-time information generated by ongoing experiments.
“Somewhat ironically, the centralizing of our experimentation infrastructure is what makes our organizational decentralization possible,” Vermeer explained to me. “Everyone uses the same tools. This fosters trust in each other’s data and enables discussion and accountability. While some companies, like Microsoft, Facebook, and Google, may be more technically advanced in areas like machine learning, our use of simple A/B tests makes us more successful in getting all people involved; we have democratized testing throughout the organization.”
Just as anyone can launch an experiment, anybody can stop one.
Be Ethically Sensitive
When contemplating new experiments, companies must think carefully about whether users would consider the tests to be unethical. While the answer isn’t always clear-cut, organizations that fail to examine this question risk sparking a backlash.
Embrace a Different Leadership Model
By democratizing experimentation and following test results where they lead, companies can enable employees to make good decisions on their own and accelerate innovation and improvements. But if most decisions are made this way, what’s left for senior leaders to do
Set a grand challenge that can be broken into testable hypotheses and key performance metrics.
Employees need to see how their experiments support an overall strategic goal.
Put in place systems, resources, and organizational designs that allow for large-scale experimentation.
Scientifically testing nearly every idea requires infrastructure: instrumentation, data pipelines, and data scientists. Several third-party tools and services make it easy to try experiments, but to scale things up, senior leaders must tightly integrate the testing capability into company processes.
Be a role model.
Leaders have to live by the same rules as everyone else and subject their own ideas to tests.
Recognize that words alone won’t change behavior.
Ultimately, being a leader in an experiment-driven organization means letting go and empowering employees to perform their own tests—which doesn’t happen by simply telling people that they can do so. It requires a concerted effort like IBM’s.