No. 145 of 500 No. 15 in Hands-On AI & Programming

Trustworthy Online Controlled Experiments

A Practical Guide to A/B Testing

By Ron Kohavi, Diane Tang and Ya Xu

Published
April 2, 2020
Print length
290 pages
Reading level
Intermediate
ISBN-13
978-1108724265
Publisher
Cambridge University Press
Written for
Data scientists, ML engineers and product leaders who run or interpret online experiments.

Synopsis

Ron Kohavi, Diane Tang and Ya Xu built and ran experimentation programs at Microsoft, Google and LinkedIn, companies that each run more than 20,000 controlled experiments a year, and this book distills what they learned about getting numbers that can be believed. Part I covers the fundamentals: why online controlled experiments are the most reliable way to establish causality, how to design and analyze a test, and Twyman's law, the principle that any figure that looks interesting or different is usually wrong. Later parts explain why page speed matters, how to choose organizational metrics and an overall evaluation criterion, how to build institutional memory, the ethics of experimenting on users, and which observational methods can substitute when randomization is impossible.

The advanced sections address platform engineering, including client-side experiments, instrumentation, choice of randomization unit, ramping exposure and scaling analysis, followed by the statistics of variance estimation, sensitivity improvements and analytical pitfalls such as sample ratio mismatch and carryover effects. Chapters draw on real experiments, including counterintuitive results from Bing and other large sites. For AI teams, the book supplies the evaluation discipline that offline metrics cannot: a way to show that a new ranking model, recommender or LLM feature improves user outcomes in production.

Scorecard

Editorial score

How it scored

Editorial score: 84.6 / 1005 weighted criteria

  • Influence30%

    25.5/30
  • Authority25%

    23.8/25
  • Practical value20%

    17.0/20
  • Currency15%

    11.3/15
  • Readability10%

    7.0/10
  • Editorial score

    84.6/100

How the rank was computed

Rank No. 145 of 500 comes from a composite of 59.9: observable signals 21.7 of 55 points plus editorial review 38.2 of 45 points.

All 11 inputs
InputPoints
Observable signals21.7/55
Reach (Amazon ratings volume)10.1/16.5
Consensus across best-of lists2.8/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted7.3/8.3
Amazon sales rank1.5/5.5
Recency0.0/2.8
Editorial review38.2/45
Author authority10.7/11.3
Influence11.5/13.5
Practical value7.7/9.0
Readability3.2/4.5
Currency5.1/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

The standard practitioner's reference for running A/B tests that produce trustworthy results, essential for anyone evaluating ML-driven product changes.

Strengths

  • Authors ran experimentation at Microsoft, Google, LinkedIn
  • Pairs statistical rigor with organizational and cultural advice
  • Packed with real experiments and documented pitfalls

Keep in mind

It focuses on web and app experimentation at scale; teams with small traffic or offline ML evaluation needs will find parts less applicable.

Key takeaways

  1. Twyman's law applies: a surprising experimental result is more likely a bug or instrumentation error than a breakthrough, so validate it before acting.
  2. Define an overall evaluation criterion tied to long-term user value, because optimizing short-term metrics such as clicks can damage the product.
  3. Trust checks such as sample ratio mismatch tests catch broken experiments that would otherwise yield confident but wrong conclusions.
  4. Most ideas fail to improve their target metrics when tested, so organizations that experiment at scale learn faster than those relying on intuition.

Notable facts

About the author

Ron Kohavi, Diane Tang and Ya Xu

Ron Kohavi was a technical fellow and corporate vice president at Microsoft, where he founded the Experimentation Platform team, then vice president and technical fellow at Airbnb; earlier he directed data mining and personalization at Amazon. He holds a Stanford PhD. Co-authors are Diane Tang, a Google Fellow, and Ya Xu, head of data science and experimentation at LinkedIn when the book appeared.

Questions

What is Trustworthy Online Controlled Experiments about?
It is a practical guide to A/B testing written by experimentation leaders from Microsoft, Google and LinkedIn. It explains how to design and analyze online controlled experiments, choose metrics, build an experimentation platform and culture, and detect the many ways results can mislead. Advanced chapters cover randomization units, ramping, variance estimation and observational alternatives when experiments are impossible.
Why is an A/B testing book relevant to AI?
Machine learning models are usually judged offline on held-out data, but only a controlled experiment shows whether a new model changes user behavior or business results. Recommendation, search ranking and LLM-powered features are routinely launched through A/B tests, and this book explains how to run those tests so that their conclusions are reliable.
Is Trustworthy Online Controlled Experiments worth reading in 2026?
Yes. The statistical and organizational principles it teaches have not changed since its 2020 publication, and its endorsers include senior figures at Google, Microsoft and Netflix. It does not address evaluating generative AI specifically, so teams testing LLM features should combine its methods with newer guidance on offline evaluation, but the core discipline of trustworthy experimentation applies directly.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.