Technical usage and inference docs: TECHNICAL_README.md
✨ Overview
W1-dLLM is a diffusion language model.
It is built on a simple idea that is now becoming increasingly clear:
Powerful language modeling does not have to be autoregressive.
And today, for the first time, that idea feels like more than a hypothesis.
We are seeing a remarkably meaningful set of results emerge together:
- stable optimization
- genuine scaling on variable-length text
- strong self-revision and error-correction behavior
- clear generalization in a relatively small model
- and an increasingly strong signal that diffusion language models are moving beyond their old "semi-autoregressive" transitional phase
What is beginning to emerge is something more fundamental:
True parallel generation.
This model does not merely generate text.
It behaves more like a system that first forms, then revises, then refines its expression in latent space before finally settling it into language.
At one mid-training checkpoint, when the model first showed this behavior, the entire company gathered around the same screen as if witnessing the birth of a newborn.
And that moment mattered.
🧠 Model Architecture
W1-dLLM uses a diffusion Transformer architecture for language modeling rather than a standard autoregressive decoder.
Core Configuration
- 48 diffusion Transformer blocks
- 50 layers total if counting the input embedding layer and final output layer
- Vocabulary size: 64,512
- Hidden size: 2,048
- Attention inner dimension: 3,072
- FFN dimension: 7,168
Architectural Ingredients
- timestep embeddings
- rotary positional encoding (RoPE)
- adaptive LayerNorm modulation
- RMSNorm
- SwiGLU feed-forward networks
What We Believe Matters Most
1. The diffusion Transformer structure itself is essential
This is not a cosmetic wrapper change. It is the foundation that makes these capabilities possible.
2. Adaptive LayerNorm modulation is critical
Without it, conditional control in the diffusion process and training stability both degrade significantly.
3. Some classic autoregressive Transformer optimizations transfer well
For example, increasing the attention inner dimension remains beneficial.
4. Bidirectional attention must be designed in from pretraining
Bidirectionality cannot simply be patched in at inference time.
If one wishes the model to truly think, revise, and converge in parallel, that capability must be baked into both the architecture and the pretraining setup from the start.
⚙️ Training Characteristics
One of the clearest engineering conclusions from this training run is:
Diffusion language models require sufficiently large batch sizes.
In our training, once batch size exceeded 500, the model's training behavior, stability, and efficiency all improved noticeably.
This differs from many familiar autoregressive training heuristics and suggests that diffusion training has its own scaling regime.
Hardware Utilization
We also observed model FLOP utilization remaining around 45% throughout training.
That number matters.
It shows that this path is not merely theoretically parallel — it is already demonstrating real engineering value on actual hardware.
For a large-batch, strongly parallel diffusion language model, 45% MFU is already meaningful.
And it is clearly not the upper bound.
Optimization Behavior
Training loss declined continuously throughout the run, and did so with unusual stability:
- no collapse
- no rollback
- no obvious instability event
More importantly, even when continuing to train on the same data, the loss still kept going down.
That gives us a strong signal:
Diffusion-style training may significantly alleviate data exhaustion.
High-quality data remains extremely important — perhaps even more so for smaller diffusion language models.
But what we are seeing now is this:
High-quality data is not only valuable — it may also be reusable more fully and more repeatedly.
🔍 What We Found
1️⃣ Latent-space thinking is not an illusion — it is real
From the beginning of pretraining, we intentionally encouraged the model to learn a more global style of latent-space reasoning.
What emerged was not merely a more convincing surface-level chain of thought.
Before expressing anything, the model appears to genuinely organize, revisit, and reconstruct its own ideas.
Even more excitingly, we observed the model briefly switching between different languages during this process:
- English
- Korean
- Japanese
- Chinese
- and sometimes symbolic fragments
These appear transiently and then gradually settle into a final language form.
This does not look like simple multilingual mixing.
It looks more like the model is searching for a better internal representation before compressing it into final visible language.
The model's thinking is not the same thing as its final output text.
2️⃣ It does not just continue text — it revises while generating in parallel
One of the most exciting observations from this run was the model's strong ability to:
- revise
- correct itself
- reflect
- converge toward better answers
But the important part is that this revision happens inside the parallel generation process itself.
This is not linear generation followed by a repair pass.
Instead, the model seems to expand globally while continuously revising and converging on its current predictions.
During benchmarking, we often saw the model:
- delay final answer commitment until late
- repeatedly refine answer candidates
- repair its final answer choice near the end
This feels fundamentally different from many earlier systems that were still semi-autoregressive underneath a diffusion shell.
Our conclusion is becoming increasingly clear:
The power of bidirectional attention cannot be recovered only at inference time.
It has to be designed jointly with the model architecture from the pretraining stage onward.
Once this is done correctly, what emerges is no longer semi-autoregression.
It is much closer to:
True parallel diffusion generation.
3️⃣ We really made flexible-length text work
Training a high-quality diffusion language model is already difficult.
Handling flexible-length text well is even harder.
Different prompts should naturally lead to responses of different lengths.
So length is not a peripheral variable.
It is one of the core variables of quality itself.
Once we started treating length as a first-class modeling problem, the model's flexible-length generation capability improved significantly.
Length is not a side detail — it is a core modeling problem.
4️⃣ Small models can generalize too — and that matters a lot
Even at relatively modest scale, the model still demonstrated clear generalization ability.
And with only a small amount of supervised fine-tuning data, it could be steered toward the behaviors we wanted.
This changes the deployment story directly.
It suggests that diffusion language models may not be useful only for research.
They may also fit a very practical path:
- fast
- cheap
- usable
5️⃣ Parallelism is finally more than a slogan
In the past, many diffusion language model systems were still semi-autoregressive in essence.
This time feels different.
What we see now is a much stronger and clearer signal of real parallel generation.
And this parallelism is not simply about producing multiple tokens at once.
It means three things happening at the same time:
- generate simultaneously
- revise simultaneously
- converge simultaneously
Simultaneously generate. Simultaneously revise. Simultaneously converge.
This directly affects:
- throughput
- latency
- cost
- product viability
In a world of exploding token consumption, that becomes especially compelling.
6️⃣ Self-distillation is not imaginary
We increasingly believe that self-distillation will be a key next step in unlocking even more of this paradigm's parallel potential.
The level of parallelism we see today is still far from the ceiling.
Through self-distillation, we may be able to:
- reduce the number of diffusion steps
- process more tokens per step
- lower the total number of steps needed for generation
- improve throughput while preserving quality
In practical terms, that means:
Bigger steps, fewer total steps, faster generation, cheaper tokens.
And the significance goes beyond inference cost alone.
It suggests that the efficiency curve of diffusion language models is still far from fully explored.
The speed we see today is not yet the true speed of this paradigm.
🌍 Why This Matters
For a long time, diffusion language models were often treated as:
- an interesting alternative
- a research curiosity
- or merely a different decoding strategy
But these results point to something deeper:
This may truly be a new language modeling paradigm.
What matters is not one isolated metric.
It is the rare possibility that many advantages may hold at the same time:
- cheaper, more stable training dynamics
- more efficient GPU batch utilization
- stronger throughput
- lower time-to-first-token latency
- cheaper per-token pricing
- a natural path toward latent-space thinking
- a natural path toward self-revision
- more flexible modeling
- a large post-training optimization space still waiting to be explored
To put it plainly:
The value of diffusion language models is no longer philosophical.
It is becoming an engineering reality.
🖼️ Toward Broader Fusion
We are also actively exploring larger-scale fused diffusion models across text, images, and beyond.
We do not believe diffusion-based modeling is limited to language alone.
It may also provide an alternative path for embodied intelligence systems built on vision-language-action foundations, especially in robotics.
So what we are seeing today is not the end.
It may not even be the real beginning yet.
⚠️ Limitations
We also want to stay clear-eyed about where things stand.
This system does not mean diffusion models have already replaced autoregressive models across every dimension.
The most exciting capabilities are still early:
- latent-space reasoning
- iterative self-revision
- unified cross-modal diffusion modeling
All of these still lack sufficiently mature and standardized evaluation frameworks.
So we should remain ambitious, but also honest.
Even so, the signal is already strong enough for us to say this clearly:
The upper bound of diffusion language models is still far from truly visible.
💥 What We Believe
We believe we are working on something genuinely important.
We want to drive token prices down dramatically and bake fast, cheap, and useful into the model's DNA.
There is a market for being a ruthless price disruptor.
And there is still so much left unexplored.
🤝 Join Us
If you also value:
- honesty and reliability
- deep curiosity
- the courage to explore the unknown
we would love for you to consider joining us.
At Whaletech, we have:
- abundant GPU resources
- fast iteration speed
- exceptional and deeply committed teammates
- an intensely energetic technical culture
We look forward to exploring a much larger frontier together — one that has not yet been fully defined.