“RL Is Not The Full Story”: Former Tesla AI Chief Andrej Karpathy
Humans engage in reflective thinking — reviewing what worked, what didn’t, and why — then extracting explicit lessons that can be verbalized and applied in future situations
Scaling up reinforcement learning (RL) has become a major focus in AI development, and while it offers significant potential, it may not be the complete solution for advancing AI systems. At its core, RL adjusts action probabilities based on outcomes — reinforcing actions that lead to success and discouraging those that don’t. This method is powerful because it leverages scalar rewards to guide learning without requiring detailed supervision. However, as tasks grow more complex and require longer sequences of interactions, relying solely on this feedback mechanism becomes inefficient. Learning from a single number at the end of a lengthy process seems increasingly impractical.
RL doesn’t align well with how humans typically learn and improve. Humans engage in reflective thinking — reviewing what worked, what didn’t, and why — then extracting explicit lessons that can be verbalized and applied in future situations. These insights often take the form of new instructions or strategies, which over time become second nature through repetition and internalization, somewhat like how sleep helps consolidate learning. Current RL methods lack this kind of structured reflection phase, which could allow models to accumulate and utilize richer, more explicit knowledge from each experience.
An example of such a reflective system might involve running multiple task rollouts, analyzing them together using a meta-prompt to extract key lessons, and then incorporating those insights into the model’s system prompt or a dynamic lesson database. While early features like ChatGPT’s Memory hint at this direction, they are currently limited to customization rather than problem-solving. Traditional RL environments, such as Atari games, lack the linguistic and contextual flexibility needed for such an approach since they don’t involve large language models or in-context learning.
A practical illustration of how reflection can help is seen in how Claude was modified to handle letter-counting tasks — an issue LLMs struggle with due to tokenization and processing limitations. By explicitly instructing the model to separate letters and count them step-by-step, performance improved significantly. The challenge now is to develop systems where such lessons emerge organically through agent experience, rather than being manually engineered, and to refine and distill these lessons over time to avoid bloating context usage.
While RL will continue to drive progress — especially when implemented effectively — it likely isn’t the final paradigm for building advanced AI systems. As rollout lengths increase and tasks become more sophisticated, alternative learning mechanisms inspired by human cognition, particularly those involving reflection and explicit knowledge accumulation, may unlock new S-curves of capability. These approaches could be especially impactful in domains involving large language models, where traditional RL techniques fall short, offering exciting possibilities for future research and innovation.
Reflection and Planning as Humans Do
Recently, decomposing complex problems into simple subtasks–a crucial part of human-like natural planning–to solve the given problem has significantly boosted the performance of large language models (LLMs). However, leveraging such planning structures during post-training to boost the performance of smaller open-source LLMs remains underexplored. Motivated by this, we introduce PLANTUNING, a unified post-training framework that (i) distills synthetic task decompositions (termed “planning trajectories”) from largescale LLMs and (ii) fine-tunes smaller models via supervised and reinforcement-learning objectives designed to mimic these planning processes to improve complex reasoning. On GSM8k and the MATH benchmarks, plantuned models outperform strong baselines by an average ∼ 7%. Furthermore, plan-tuned models show better generalization capabilities on out-of-domain datasets, with average ∼ 10% and ∼ 12% performance improvements on OlympiadBench and AIME 2024, respectively. Our detailed analysis demonstrates how planning trajectories improves complex reasoning capabilities, showing that PLANTUNING is an effective strategy for improving task-specific performance of smaller LLMs.
We introduce PLAN-TUNING, a novel posttraining method that incorporates synthetic natural planning trajectories into smaller LLMs’ parameters, rather than relying solely on inference-time prompts. We develop two complementary posttraining strategies — SFT to imitate high-quality plan decompositions and GRPO to reinforce plan quality alongside answer correctness — thereby teaching models both how to plan and how to execute. Across two in-domain datasets (GSM8K, MATH), PLAN-TUNED models achieve an average ∼ 7% accuracy boost over strong baselines; on out-of-domain benchmarks (OlympiadBench, AIME), they significantly improved the performance. Through detailed analyses, we show (i) that a good plan relies on dataset-specific consistency — mixing heterogeneous sources degrades performance, and (ii) that plan-tuning substantially mitigates the token-repetition issue.
