No. 208 of 500 No. 35 in Machine Learning

Information Theory, Inference, and Learning Algorithms

By David J. C. MacKay

Published
September 25, 2003
Print length
642 pages
Reading level
Advanced
ISBN-13
978-0521642989
Publisher
Cambridge University Press
Written for
Graduate students, researchers and engineers wanting deep probabilistic foundations for machine learning and coding.

Synopsis

David MacKay's textbook rests on a conviction that information theory and machine learning are two faces of one subject, since both concern reasoning under uncertainty and representing data efficiently. The book begins with probability, entropy and Shannon's theorems, then develops data compression, including Huffman and arithmetic coding, and noisy-channel coding before branching into broader topics in information theory. The large middle section on probabilities and inference builds a Bayesian toolkit, covering maximum likelihood, clustering, Laplace's method, Monte Carlo methods, variational approximations, message passing and model comparison through Occam's razor.

A section on neural networks treats single neurons, Hopfield networks, Boltzmann machines and Gaussian processes as inference machines, and the closing chapters explain sparse-graph codes such as low-density parity-check, turbo and digital fountain codes. More than 400 exercises, some with detailed solutions, and digressions into crosswords, codebreaking and the evolutionary value of sex keep the tone conversational. The neural network material predates deep learning, so readers will not find convolutional networks or transformers here. What they gain is a durable probabilistic way of thinking that underlies much of modern generative modeling, from variational autoencoders to the view of language modeling as compression. Cambridge University Press still sells the print edition, and MacKay's site hosts the full text for on-screen reading.

Scorecard

Editorial score

How it scored

Editorial score: 79.8 / 1005 weighted criteria

  • Influence30%

    25.5/30
  • Authority25%

    23.8/25
  • Practical value20%

    13.0/20
  • Currency15%

    10.5/15
  • Readability10%

    7.0/10
  • Editorial score

    79.8/100

How the rank was computed

Rank No. 208 of 500 comes from a composite of 56.4: observable signals 20.4 of 55 points plus editorial review 36.0 of 45 points.

All 11 inputs
InputPoints
Observable signals20.4/55
Reach (Amazon ratings volume)8.4/16.5
Consensus across best-of lists5.5/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted5.4/8.3
Amazon sales rank1.1/5.5
Recency0.0/2.8
Editorial review36.0/45
Author authority10.7/11.3
Influence11.5/13.5
Practical value5.9/9.0
Readability3.2/4.5
Currency4.7/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

A lucid, freely readable textbook that unites information theory, Bayesian inference and neural networks; demanding, but rewarding for mathematically prepared readers.

Strengths

  • Unifies information theory with Bayesian machine learning
  • More than 400 exercises for self-study
  • Full text free to read online

Keep in mind

It demands comfort with probability and calculus, and its neural network chapters predate deep learning, so readers seeking modern architectures must look elsewhere.

Key takeaways

  1. Information theory and machine learning share foundations, because compressing data well and predicting it well are closely linked problems.
  2. Bayesian inference offers one framework for learning, model comparison and uncertainty, with Occam's razor emerging naturally from probability theory rather than as an added rule.
  3. When exact inference is intractable, Monte Carlo methods, variational approximations and message passing provide practical approximate alternatives.
  4. Sparse-graph codes such as low-density parity-check and turbo codes approach Shannon's limit and show inference algorithms at work in communications engineering.

Notable facts

  • Caltech's Bob McEliece called it 'an instant classic,' and Zoubin Ghahramani and Peter Dayan of UCL's Gatsby Unit called it 'an extraordinary and important book.' Source (opens in a new tab)
  • MacKay kept the complete book available for on-screen reading on his website after Cambridge University Press published it in September 2003. Source (opens in a new tab)

About the author

David J. C. MacKay

Sir David J. C. MacKay (1967-2016) was Regius Professor of Engineering at the University of Cambridge and chief scientific adviser to the UK Department of Energy and Climate Change from 2009 to 2014. He earned his PhD at Caltech under John Hopfield, was elected a Fellow of the Royal Society in 2009, invented the Dasher text-entry system and wrote Sustainable Energy - Without the Hot Air.

Questions

Is Information Theory, Inference, and Learning Algorithms free?
The full text can be read on screen at no cost on David MacKay's website, where he posted PDF and other versions alongside the published edition. Cambridge University Press holds the copyright and sells the printed book, and the site states that copying the whole book onto paper is not permitted. Many readers start online and buy the print copy for sustained study.
Is MacKay's book still relevant for machine learning in 2026?
Yes, for its foundations. Its treatment of entropy, Bayesian inference, Monte Carlo and variational methods remains central to probabilistic machine learning and generative modeling. Its neural network chapters cover Hopfield networks, Boltzmann machines and Gaussian processes rather than convolutional networks or transformers, so readers should pair it with a modern deep learning text for current architectures.
Who should read Information Theory, Inference, and Learning Algorithms?
It suits advanced undergraduates, graduate students, researchers and engineers comfortable with probability, linear algebra and calculus who want to understand why learning algorithms work, not just how to call them. Self-learners benefit from its 400-plus exercises. Executives and beginners seeking an introduction to AI will find it too mathematical and should start with an accessible overview instead.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.