Diffusion LLM // Finally it is working
Diffusion approach is used for images, video, and music // now hacked for text and code
Actually data is a key, models are just wrappers | compressors | approximators around it.
Most LLMs generate tokens autoregressively, meaning they generate tokens from left to right.
Image and video models are diffusion models, they generate an entire output at once and then gradually denoise it. [link]
Despite their groundbreaking performance for many generative modeling tasks, diffusion models have fallen short on discrete data domains such as natural language. Crucially, standard diffusion models rely on the well-established theory of score matching, but efforts to generalize this to discrete structures have not yielded the same empirical gains. In this work, we bridge this gap by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build discrete diffusion models, and significantly boosts performance. Experimentally, we test our Score Entropy Discrete Diffusion models (SEDD) on standard language modeling tasks. For comparable model sizes, SEDD beats existing language diffusion paradigms (reducing perplexity by 25–75%) and is competitive with autoregressive models, in particular outperforming GPT-2. Furthermore, compared to autoregressive mdoels, SEDD generates faithful text without requiring distribution annealing techniques like temperature scaling (around 6- 8× better generative perplexity than un-annealed GPT-2), can trade compute and quality (similar quality with 32× fewer network evaluations), and enables controllable infilling (matching nucleus sampling quality while enabling other strategies besides left to right prompting).
Mercury is the first commercial-scale language model based on the diffusion paradigm, traditionally used for images, video, and music, now adapted for text and code generation. This new approach offers significant speed improvements (up to 10x faster) and high-quality outputs compared to traditional autoregressive models. Key highlights include:
- Speed: Achieves over 1000 tokens/second on NVIDIA H100, making it up to 10x faster than traditional models.
- Quality: Matches or exceeds speed-optimized models like GPT-4o-mini and Claude 3.5 Haiku on coding benchmarks.
- Rankings: Ranked #1 in speed and #2 in quality on Copilot Arena, outperforming models like GPT-4o and Gemini-1.5-Flash.
Mercury’s diffusion-based LLMs (dLLMs) are already being adopted by early users, improving user experiences and reducing costs, especially in latency-sensitive applications. The models are compatible with existing hardware, datasets, and fine-tuning pipelines.
