No. 229 of 500 No. 26 in Generative AI & LLMs

Hands-On Generative AI with Transformers and Diffusion Models

By Omar Sanseviero, Pedro Cuenca, Apolinário Passos and Jonathan Whitaker

Print length
416 pages
Publisher
O'Reilly Media
Reading level
Intermediate
ISBN-13
978-1098149246
Published
December 31, 2024
Written for
Developers and ML practitioners who want to understand, fine-tune and combine open generative models.

Synopsis

Published by O'Reilly in late 2024, this book explains how modern generative models for text, images and audio work and how to adapt them with open-source tools. It opens with an overview of generative media, then covers transformers, the compression and representation of information with autoencoders and embeddings, and diffusion models. Later chapters treat Stable Diffusion and conditional generation, fine-tuning language models, fine-tuning Stable Diffusion, creative applications of text-to-image models such as image editing, and audio generation, with an appendix on building an end-to-end retrieval-augmented generation pipeline.

Each chapter pairs a conceptual explanation with runnable code, which the authors publish in a GitHub repository with Google Colab links, and the examples rely on Hugging Face libraries such as transformers and diffusers. Exercises ask readers to extend models rather than simply rerun them. The authors discuss when a pretrained model suffices and when fine-tuning, including parameter-efficient methods such as LoRA, earns its cost. Readers gain working fluency with the open-model ecosystem and enough theory to follow new research papers, though those focused on proprietary frontier models or production deployment will need other sources.

Scorecard

Editorial score

How it scored

Editorial score: 69.8 / 1005 weighted criteria

  • Influence30%

    15.0/30
  • Authority25%

    20.0/25
  • Practical value20%

    16.0/20
  • Currency15%

    11.3/15
  • Readability10%

    7.5/10
  • Editorial score

    69.8/100

How the rank was computed

Rank No. 229 of 500 comes from a composite of 55.6: observable signals 24.1 of 55 points plus editorial review 31.5 of 45 points.

All 11 inputs
InputPoints
Observable signals24.1/55
Reach (Amazon ratings volume)5.4/16.5
Consensus across best-of lists11.0/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted5.1/8.3
Amazon sales rank0.9/5.5
Recency1.7/2.8
Editorial review31.5/45
Author authority9.0/11.3
Influence6.8/13.5
Practical value7.2/9.0
Readability3.4/4.5
Currency5.1/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

A clear, code-rich guide to how open generative models work and how to fine-tune them, written largely by Hugging Face practitioners.

Strengths

  • Runnable notebooks with Colab links
  • Covers text, image and audio generation
  • Authors work on the tools they teach

Keep in mind

It assumes Python and some deep learning background, needs GPU access for many examples, and gives limited attention to production deployment.

Key takeaways

  1. Transformers and diffusion models share common building blocks, and understanding tokenization, embeddings and attention makes both families easier to adapt.
  2. Pretrained open models cover many tasks; fine-tuning, often with parameter-efficient methods such as LoRA, pays off when data or style needs are specific.
  3. Diffusion models generate images by learning to reverse a gradual noising process, and conditioning signals such as text prompts steer the output.
  4. Combining existing components, such as text encoders, latent autoencoders and noise schedulers, can produce new creative applications without training from scratch.

Notable facts

  • Companion code for all nine chapters, plus an appendix on end-to-end retrieval-augmented generation, is published in the genaibook GitHub repository with Colab links. Source (opens in a new tab)

About the author

Omar Sanseviero, Pedro Cuenca, Apolinário Passos and Jonathan Whitaker

Omar Sanseviero led developer advocacy engineering at Hugging Face after working as a Google engineer on Google Assistant and TensorFlow Graphics. Coauthors Pedro Cuenca and Apolinário Passos worked as Hugging Face machine learning engineers on diffusion software and on machine learning for art, and Jonathan Whitaker is a generative-modeling researcher who wrote Hugging Face's diffusion models class.

Questions

What do I need to know before reading Hands-On Generative AI with Transformers and Diffusion Models?
You should be comfortable with Python and familiar with basic machine learning ideas such as training, loss functions and neural networks. The book explains transformer and diffusion internals from the ground up, but it moves quickly. Many examples are computationally intensive, so access to a GPU, such as a free Google Colab instance, makes the notebooks practical to run.
Does the book cover large language models like ChatGPT?
It covers the transformer architecture behind models such as ChatGPT and shows how to fine-tune open language models, including with parameter-efficient methods. Its focus is the open-source ecosystem built around Hugging Face libraries rather than proprietary APIs. Readers who mainly want to build applications on commercial chat models will find more targeted guidance in books on AI engineering and prompt design.
Is Hands-On Generative AI worth reading in 2026?
Yes, for practitioners who want to understand how generative models work rather than only call them through an API. The fundamentals it teaches, including attention, latent representations, diffusion sampling and fine-tuning, remain the basis of current systems. Specific model checkpoints and library versions have changed since late 2024, so expect to adapt some code to current releases.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.