Trustworthy Online Controlled Experiments
A Practical Guide to A/B Testing
Synopsis
Ron Kohavi, Diane Tang and Ya Xu built and ran experimentation programs at Microsoft, Google and LinkedIn, companies that each run more than 20,000 controlled experiments a year, and this book distills what they learned about getting numbers that can be believed. Part I covers the fundamentals: why online controlled experiments are the most reliable way to establish causality, how to design and analyze a test, and Twyman's law, the principle that any figure that looks interesting or different is usually wrong. Later parts explain why page speed matters, how to choose organizational metrics and an overall evaluation criterion, how to build institutional memory, the ethics of experimenting on users, and which observational methods can substitute when randomization is impossible.
The advanced sections address platform engineering, including client-side experiments, instrumentation, choice of randomization unit, ramping exposure and scaling analysis, followed by the statistics of variance estimation, sensitivity improvements and analytical pitfalls such as sample ratio mismatch and carryover effects. Chapters draw on real experiments, including counterintuitive results from Bing and other large sites. For AI teams, the book supplies the evaluation discipline that offline metrics cannot: a way to show that a new ranking model, recommender or LLM feature improves user outcomes in production.
Scorecard
Editorial score
How it scored
Editorial score: 84.6 / 1005 weighted criteria
- 25.5/30
Influence30%
- 23.8/25
Authority25%
- 17.0/20
Practical value20%
- 11.3/15
Currency15%
- 7.0/10
Readability10%
- 84.6/100
Editorial score
How the rank was computed
Rank No. 145 of 500 comes from a composite of 59.9: observable signals 21.7 of 55 points plus editorial review 38.2 of 45 points.
All 11 inputs
| Input | Points |
|---|---|
| Observable signals | 21.7/55 |
| Reach (Amazon ratings volume) | 10.1/16.5 |
| Consensus across best-of lists | 2.8/13.8 |
| Awards and bestseller evidence | 0.0/8.3 |
| Reader rating, sample-adjusted | 7.3/8.3 |
| Amazon sales rank | 1.5/5.5 |
| Recency | 0.0/2.8 |
| Editorial review | 38.2/45 |
| Author authority | 10.7/11.3 |
| Influence | 11.5/13.5 |
| Practical value | 7.7/9.0 |
| Readability | 3.2/4.5 |
| Currency | 5.1/6.8 |
Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.
Verdict
The standard practitioner's reference for running A/B tests that produce trustworthy results, essential for anyone evaluating ML-driven product changes.
Strengths
- Authors ran experimentation at Microsoft, Google, LinkedIn
- Pairs statistical rigor with organizational and cultural advice
- Packed with real experiments and documented pitfalls
Keep in mind
It focuses on web and app experimentation at scale; teams with small traffic or offline ML evaluation needs will find parts less applicable.
Key takeaways
- Twyman's law applies: a surprising experimental result is more likely a bug or instrumentation error than a breakthrough, so validate it before acting.
- Define an overall evaluation criterion tied to long-term user value, because optimizing short-term metrics such as clicks can damage the product.
- Trust checks such as sample ratio mismatch tests catch broken experiments that would otherwise yield confident but wrong conclusions.
- Most ideas fail to improve their target metrics when tested, so organizations that experiment at scale learn faster than those relying on intuition.
Notable facts
- The book has been translated into Chinese, Japanese, Korean, Russian and Polish. Source (opens in a new tab)
- Endorsers include Google's Jeff Dean and entrepreneur Steve Blank, who called it the definitive text on A/B testing. Source (opens in a new tab)
- According to the book's author page, Kohavi's papers have more than 55,000 citations. Source (opens in a new tab)
About the author
Questions
What is Trustworthy Online Controlled Experiments about?
Why is an A/B testing book relevant to AI?
Is Trustworthy Online Controlled Experiments worth reading in 2026?
Related books
- Designing Data-Intensive Applications Martin Kleppmann and Chris Riccomini
- Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow Aurélien Géron
- The Data Warehouse Toolkit Ralph Kimball and Margy Ross
- R for Data Science Hadley Wickham and Garrett Grolemund
- Python Data Science Handbook Jake VanderPlas
- Fundamentals of Data Engineering Joe Reis and Matt Housley
Sources
As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.






