Grokking Deep Reinforcement Learning
Synopsis
Miguel Morales, who works on reinforcement learning at Lockheed Martin and teaches in Georgia Tech's Reinforcement Learning and Decision Making course, builds deep reinforcement learning from first principles. The first half develops intuition with tabular methods: Markov decision processes, policy and value iteration, multi-armed bandits and the exploration problem, Monte Carlo and temporal-difference prediction, and control algorithms such as SARSA and Q-learning. The second half adds neural networks, moving from neural fitted Q-iteration through DQN, double DQN, dueling networks and prioritized experience replay, and then to policy-gradient and actor-critic methods including REINFORCE, A3C, GAE, DDPG, TD3, SAC and PPO.
Each algorithm is presented three ways: an intuitive explanation, annotated mathematics, and PyTorch code in Jupyter notebooks from a companion GitHub repository that runs inside a Docker container. A closing chapter surveys topics the book leaves out, such as model-based methods, meta-learning and multi-agent learning, and frames them as steps toward artificial general intelligence. Charles Isbell, then dean of computing at Georgia Tech, wrote the foreword. Readers finish able to implement, tune and debug standard agents and to read current reinforcement learning papers, although the 2020 text does not cover reinforcement learning from human feedback for language models.
Scorecard
Editorial score
How it scored
Editorial score: 65.3 / 1005 weighted criteria
- 16.5/30
Influence30%
- 16.3/25
Authority25%
- 16.0/20
Practical value20%
- 9.0/15
Currency15%
- 7.5/10
Readability10%
- 65.3/100
Editorial score
How the rank was computed
Rank No. 335 of 500 comes from a composite of 51.9: observable signals 22.5 of 55 points plus editorial review 29.4 of 45 points.
All 11 inputs
| Input | Points |
|---|---|
| Observable signals | 22.5/55 |
| Reach (Amazon ratings volume) | 7.3/16.5 |
| Consensus across best-of lists | 8.3/13.8 |
| Awards and bestseller evidence | 0.0/8.3 |
| Reader rating, sample-adjusted | 6.2/8.3 |
| Amazon sales rank | 0.7/5.5 |
| Recency | 0.0/2.8 |
| Editorial review | 29.4/45 |
| Author authority | 7.3/11.3 |
| Influence | 7.4/13.5 |
| Practical value | 7.2/9.0 |
| Readability | 3.4/4.5 |
| Currency | 4.1/6.8 |
Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.
Verdict
A patient, code-first introduction to deep reinforcement learning that builds from bandits and MDPs to PPO and soft actor-critic.
Strengths
- Builds intuition before introducing the mathematics
- Runnable notebooks for nearly every chapter
- Covers modern algorithms through SAC and PPO
Keep in mind
Published in 2020, it predates RLHF and large-model applications, and readers need working Python and basic deep learning experience to keep pace.
Key takeaways
- Reinforcement learning is hard because feedback is sequential, evaluative and sampled at once, unlike the immediate, labeled feedback of supervised learning.
- Exploration has a measurable cost: bandit strategies such as epsilon-greedy, softmax and upper confidence bounds trade information gathering against immediate reward.
- DQN stabilizes value-based learning with target networks and experience replay, and double DQN reduces the overestimation of action values.
- Actor-critic methods such as SAC and PPO pair a learned policy with value estimates to handle continuous actions and steadier training.
Notable facts
- Manning published the 472-page book in October 2020 with a foreword by Charles Isbell. Source (opens in a new tab)
- The companion GitHub repository, mimoralea/gdrl, has about 1,100 stars and 280 forks. Source (opens in a new tab)
About the author
Questions
What does Grokking Deep Reinforcement Learning cover?
Do I need a strong math background for Grokking Deep Reinforcement Learning?
Is Grokking Deep Reinforcement Learning still useful in 2026?
Related books
- Multi-Agent Reinforcement Learning Stefano V. Albrecht, Filippos Christianos and Lukas Schäfer
- Machine Learning Tom M. Mitchell
- Machine Learning Q and AI Sebastian Raschka
- Understanding Machine Learning Shai Shalev-Shwartz and Shai Ben-David
- Algorithms for Decision Making Mykel J. Kochenderfer, Tim A. Wheeler and Kyle H. Wray
- Pattern Classification Richard O. Duda, Peter E. Hart and David G. Stork
Sources
As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.






