Draft-Side Learning in Speculative Decoding Has Two Clocks
update at: 2026/08/15
TL;DR
Speculative decoding already reveals where a draft model diverges from its target, but turning that verification signal into request-local memory learning adds work to the decode path.
The central hypothesis is that longer or locally repetitive streams may gain enough accepted tokens to repay memory reads, gradient updates, and synchronization; no matched end-to-end result has established this yet.
Persistent draft-weight or adapter updates learn on a different clock, so their progress cannot substitute for the in-loop latency test.
Long Outputs Expose the Static Drafter's Limits
Speculative decoding lets a small drafter propose tokens and asks a larger target model to verify them. When verification applies the appropriate acceptance and correction rule, the target model still defines the output distribution. The drafter determines how many target-model calls can be saved, not what the target model knows. The original Speculative Decoding study reported a 2–3× acceleration for its tested T5-XXL setup; that number is not a universal speedup.
The original note's overview figure is retained as a map of the proposed idea, not as evidence that draft-side learning is already fast or effective.
Figure 1 — Original motivation and proposal sketch (source Figure 1). The left panel groups long-form generation, in-context learning, and task adaptation as motivations; the right proposes memory updates in the drafter after target verification. Its “free draft,” “minimal overhead,” and RNN/Diffusion support labels are hypotheses, not measured outcomes. Original image.
The interesting failure mode is a mismatch between the drafter and the target along the current stream. A drafter trained offline may predict early tokens well but become less useful as the generated context moves away from its training distribution. The TTS study reports this decline for several speculators on long outputs. The claim is workload-specific, but it exposes a real systems question: can verification feedback make later drafts better quickly enough to repay the cost of learning?
Updating a drafter online is already a research direction. online speculative decoding distills target behavior into draft models on observed queries. OnlineSPEC frames draft adaptation as online learning, and TTS adapts a speculator during generation. The proposal here is narrower: keep the target and most of the drafter fixed, and update a small, request-local memory state used by the drafter. Whether that restriction is a better speed–adaptation trade-off has not been established.
A Request-Local Memory Would Update After Verification
Let the frozen target define p(next token | prefix). Let a lightweight drafter define q(next token | prefix, memory). The memory is a compact state attached to one request, not a change to the target model and not a persistent store of verified facts. The drafter must be trained or retrofitted to use this state; attaching a state vector to a model that ignores it cannot create adaptation.
The original sketch makes the proposed placement clear, but leaves the learning objective and verification contract underspecified.
Figure 2 — Exploratory draft-side RNN memory loop (source Figure 2). The drawing places an RNN state and memory encoder before the drafter, then routes verification feedback toward an update for the next round. Its cross-entropy and gradient labels are a proposal, not a complete loss specification or a measured latency result. Original image.
One minimal cycle would be:
The drafter produces candidates and records the proposal probabilities used for that round.
The target verifies those candidates using ordinary speculative-decoding correction. Its token distributions, where available, become a teacher signal.
After the round commits its output tokens, update only the request-local memory by reducing the mismatch between the frozen target distribution and the drafter distribution on valid, aligned positions.
Use the updated memory in the next draft round. Reset or isolate it at the request boundary.
This separates three roles that the original draft blurred. A recurrent state update can summarize accepted context without gradients. A gradient step on that state is test-time learning. Updating the drafter’s weights is a larger online-distillation intervention, as explored in prior work. These should be separate baselines, not interchangeable names for the same mechanism.
Verification supplies useful feedback, but a gradient update is not free. It may require retaining or recomputing draft activations, reading target probabilities, launching backward kernels, and synchronizing before the next dependent draft. If the verifier exposes only an accept/reject bit, the learning signal is weaker than a full target distribution. The target need not be backpropagated through, yet the draft-side update still sits on an end-to-end latency budget.
The target-distribution contract also needs explicit care. For each round, the verifier must use the same proposal distribution that generated its candidates and apply the exact correction rule. Updating memory after the round is safer than changing the proposal midway through verification. Distribution-preservation should be checked in tests rather than assumed from the phrase “target remains frozen.”
Memory State and LoRA Weights Solve Different Adaptation Problems
A request-local state may adapt to recurring names, formats, or local structure without changing a shared drafter for every user. It also has a clear reset and isolation boundary. This is a hypothesis about the location and granularity of adaptation, not evidence that state updates are cheaper or better.
The distinction matters when comparing with LoRA. LoRA freezes base-model weights and trains small low-rank matrices; it does not update “all model weights,” as the original comparison suggested. Its cost depends on where the adapters are placed and which activations must be retained. A fair experiment could put a LoRA adapter on the drafter, not only on the target, and compare it with a memory-state update under the same total compute budget. The two techniques answer different questions: changing reusable weights versus changing state for one stream.
The original comparison graphic is preserved so its assumptions can be examined, but it is not a measured or fair head-to-head result.
Figure 3 — Original memory-versus-LoRA comparison (source Figure 4). The chart places memory on the drafter and LoRA on the main model, then depicts LoRA as updating all weights and memory updates as negligible. Those are not general properties of LoRA or measured costs; a fair baseline would place both interventions on the drafter and include their full update time. Original image.
Titans provides an important example of neural memory learned at test time, including long-context experiments. It does not demonstrate that its memory can be inserted into a speculative drafter at negligible cost. That connection is motivation for this design, not a reported result.
A Better Drafter Is Not Necessarily a Faster Decoder
Better draft probabilities do not automatically mean faster decoding. The relevant inequality is simple: time saved by avoiding target-model work must exceed memory-read, update, and synchronization time. Acceptance length is an intermediate signal; end-to-end latency and throughput decide whether the system is useful.
The first experiment should use one frozen target and one fixed drafter, then compare:
Static speculative decoding with no extra memory.
The same drafter with a recurrent state but no test-time gradient.
The same drafter with request-local memory updated from verification feedback.
An online draft-weight or draft-side LoRA baseline under a matched update budget.
Measure accepted tokens per verification, target calls, time to first token, decode latency, throughput at several batch sizes, GPU memory, and the memory-update time itself. Vary prompt and output length, especially long outputs where drafter mismatch is plausible. Keep hardware, draft length, sampling settings, and target-verification logic aligned. Test output-distribution invariance separately from acceptance gains.
For methods that retain updated draft weights or adapters across requests, also track adaptation after a workload shift, regression on earlier domains, and the time from an update to a serveable revision. Those are different costs from a gradient step on the current request's critical path.
Useful ablations include memory dimension, update frequency, reset policy, hard-token versus soft-logit supervision, and whether rejected candidate paths are excluded from the state update. A negative result is informative: if the state helps only when its update is excluded from timing, the design has learned a better predictor but not built a faster decoder.
Diffusion Drafting Does Not Validate Diffusion Memory
The original note proposed a diffusion-based latent memory and described an ordinary gradient step as “score matching.” Those are not the same operation. It also treated smoother optimization, uncertainty estimates, and rollback as inherent benefits, without an objective, sampler, or experiment establishing them. A diffusion memory might be worth testing later, but multiple denoising steps create a more demanding latency budget than the minimal recurrent-state baseline.
Figure 4 — Original diffusion-memory extension (source Figure 3). The sketch shows a latent forward process and a gradient-guided reverse step feeding the drafter. It labels the gradient step “score matching,” but no score-matching objective or sampler is specified; this is an untested extension, not evidence of smoother learning or faster decoding. Original image.
There is separate work on diffusion-based drafting, such as DiffuSpec and DART. Those results should not be presented as evidence for diffusion-based memory learning. First establish whether a small state update can beat a no-gradient baseline; only then ask whether a richer latent-state model changes the result.
Persistent Adaptation Pays Its Costs on a Different Clock
The newer work changes the comparison. “Keep learning” can mean a state that lives for one request, a drafter whose weights adapt across queries, or a policy revision that remains available after deployment. All three can use feedback, but they pay for updates on different clocks.
Mind Lab's MinT treats LoRA adapters as managed policy revisions: a shared base stays resident while adapter versions move through rollout, update, export, evaluation, serving, and rollback. This makes repeated LoRA post-training and serving an operational workflow, rather than requiring a merged full-model checkpoint for every variant. It does not show that a LoRA update is free inside a speculative decoding round, nor does adapter versioning by itself establish resistance to forgetting.
At the drafter layer, online speculative decoding already updates draft models from observed queries. The newer Draft-OPD goes after a more specific mismatch: offline supervised training sees fixed target trajectories, while inference exposes the drafter's own proposed blocks. It replays draft proposals from verification-time anchors and trains on target feedback at accepted and rejected positions; its reported serving gains are for a trained draft checkpoint. This supports on-policy draft-model training, not the claim that request-local memory updates have negligible inference overhead.
OPSD uses a different teacher construction: one model acts as student on its own rollouts and as teacher when conditioned on privileged solution information. SDFT similarly uses demonstration-conditioned self-teaching and reports sequential skill acquisition with less forgetting than supervised fine-tuning in its evaluated tasks. These results make continual learning from on-policy feedback plausible, but their privileged solutions or demonstrations are not automatically available from speculative verification, and neither paper measures this article's draft-side memory loop.
| Learning loop | What changes | How long it persists | Cost that must be counted |
|---|---|---|---|
| This proposal | Request-local draft memory | One generation, then reset | In-loop read, gradient update, and synchronization |
| Online draft training / Draft-OPD | Shared draft-model weights | Across queries and deployments | Rollout, replay, training, and model handoff |
| MinT | Versioned LoRA adapter | Across training and serving revisions | Export, evaluation, admission, and rollback |
| OPSD / SDFT | Student policy parameters | Across training tasks | Privileged supervision, update compute, and forgetting checks |
The common principle is to learn on states the system actually visits. The unresolved engineering decision is where the learned state should live. A request-local memory may react quickly without changing shared weights; a persistent drafter or adapter can amortize training over many requests. Comparing them requires both a matched feedback budget and the correct latency boundary, not a single “continuous learning” label.
The Lifetime of Learned State Sets the Trade-off
Verification feedback is an observation the pipeline already produces. Learning from it is an additional operation: it can update request-local state on the decoding critical path or feed a slower, persistent draft-policy revision. LoRA infrastructure and on-policy distillation make the latter route more practical, but they do not erase the former route's latency test. The target remains the authority on output distribution; the drafter’s memory is only an adaptive proposal state.
The falsifiable claim is therefore narrow: on sufficiently long or locally repetitive streams, a request-local memory may increase acceptance enough to outweigh its update cost. Short responses, high batching, weak teacher signals, or rapid topic shifts may erase that gain. Until a matched end-to-end experiment shows otherwise, this is a research proposal—not a demonstrated faster decoder or a more capable target model.
References
Leviathan, Yaniv, et al. “Fast Inference from Transformers via Speculative Decoding.” arXiv, 2022.
Kumar, Avinash, et al. “Test-Time Speculation.” arXiv, 2026.
Liu, Xiaoxuan, et al. “Online Speculative Decoding.” arXiv, 2023.
Qian, Yu-Yang, et al. “When Drafts Evolve: Speculative Decoding Meets Online Learning.” arXiv, 2026.
Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” arXiv, 2021.
Behrouz, Ali, et al. “Titans: Learning to Memorize at Test Time.” arXiv, 2025.
Li, Guanghao, et al. “DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding.” arXiv, 2025.
Liu, Fuliang, et al. “DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference.” arXiv, 2026.
Mind Lab. “MinT: Managed Infrastructure for Training and Serving Millions of LLMs.” arXiv, 2026.
Lei, Haodi, et al. “Draft-OPD: On-Policy Distillation for Speculative Draft Models.” arXiv, 2026.
Zhao, Siyan, et al. “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.” arXiv, 2026.
Shenfeld, Idan, et al. “Self-Distillation Enables Continual Learning.” arXiv, 2026.