Our first open-source diffusion language base model. A 48-layer diffusion Transformer with fully bidirectional attention, built diffusion-first from pretraining onward — showing stable training dynamics, true parallel generation, and clear signs of latent-space thinking and self-revision.
Bidirectional attention and parallel generation — why the two promises are hard to keep in practice.
Mask diffusion promises diffusion language models two big advantages over autoregression — parallel generation and bidirectional attention. We dissect the fundamental obstacles both promises run into in training and inference, and propose targeted training and inference remedies.