Interference-Aware DeltaNet
for Streaming Data

arXiv preprint (coming soon)

Sanghyeok Chu1, Pyunghwan Ahn†2, Gwangmo Song2, Seung Hwan Kim2,

1ECE & 4IPAI, Seoul National University 2LG AI Research 3University of Michigan

† Corresponding authors

Key Idea: We define interference as the total squared change to past-key responses, then derive a closed-form write direction that minimizes interference while preserving DeltaNet’s current-key correction.

Abstract

Transformers excel at sequence modeling through attention, but their quadratic compute and growing key-value cache make them costly for long-range streaming tasks. Linear attention variants, such as DeltaNet, address this by reformulating attention as a recurrent memory equipped with an error-corrective update rule. However, we show that DeltaNet’s update rule induces unintended interference on past-key responses, as the correction signal leaks into every past key that is not orthogonal to the current one.

We propose Interference-Aware DeltaNet (IA-DeltaNet), which generalizes the delta rule by decoupling the key used for error evaluation from the direction along which the memory is updated. We formulate the choice of write direction as a constrained quadratic program that minimizes the total squared interference on past-key responses while exactly preserving DeltaNet’s current-key correction, and derive its closed-form solution as a preconditioned key obtained from the inverse uncentered covariance of past keys. To ensure scalability, we introduce a diagonal approximation of the past-key covariance, preserving the per-step efficiency and chunk-parallelizability of the original architecture.

On multi-query associative recall, interference-aware writing improves retention under high associative load. It also improves over recurrent-memory baselines as a bounded memory in a streaming VideoLLM, and substantially mitigates DeltaNet’s degradation under prolonged recurrent-state propagation in language modeling. These results indicate that explicitly accounting for past-key geometry offers an effective design principle for associative memory in linear-attention architectures.

Interference-Aware Memory Writing

IA-DeltaNet changes the direction of a memory write to disturb past associations less, while keeping the same correction at the current key.

Interference

A write along w disturbs past responses according to its overlap with previous keys kj. We measure this as the sum of squared overlaps:

ℐt(w)=∑j<t(w⊤kj)2=w⊤Σt−1w

Σt−1 is the accumulated uncentered covariance of past keys.

Constrained optimization

Choose the write direction with the least interference, subject to preserving DeltaNet’s correction at the current unit-norm key kt:

ut=arg minwℐt(w)subject tow⊤kt=1

Closed-form solution

ut=Σt−1−1ktkt⊤Σt−1−1kt

The inverse covariance downweights directions already occupied by past keys. The denominator normalizes the write direction so that the current-key correction stays unchanged.

In practice, a diagonal covariance approximation retains DeltaNet’s per-step efficiency and chunk-parallel computation.

Associative Recall

On multi-query associative recall (MQAR), the benefit grows as more key–value associations must fit into the recurrent state. At the highest evaluated load for width 128, IA-DeltaNet improves mean accuracy from 81.7% to 95.5%; the exact variant reaches 97.5%.

MQAR accuracy (%), mean ± standard deviation over three seeds. Sequence length L = 8K.
WidthPairs KDeltaNetGated DeltaNetOSDNPDNIA-DeltaNetIA-DeltaNet†
643299.1 ±0.196.6 ±3.299.2 ±0.199.2 ±0.299.1 ±0.299.2 ±0.1
646494.2 ±2.092.6 ±8.093.4 ±5.185.4 ±21.795.1 ±0.998.2 ±0.5
6412850.7 ±23.826.2 ±23.354.3 ±6.158.3 ±12.664.0 ±0.570.0 ±5.7
1286499.3 ±0.266.2 ±57.399.3 ±0.299.1 ±0.199.2 ±0.199.4 ±0.3
12812899.1 ±0.132.8 ±56.899.2 ±0.198.7 ±0.599.1 ±0.199.6 ±0.2
12825681.7 ±14.90.0 ±0.054.8 ±19.294.3 ±2.495.5 ±1.197.5 ±2.0

† Exact IA-DeltaNet uses the full past-key covariance. Unmarked IA-DeltaNet uses the diagonal approximation throughout this page. Bold indicates the highest mean in each setting.

Recall from streaming video

In VNBench, the memory processes the entire visual stream, while the LLM sees only the last W frame readouts. IA-DeltaNet outperforms DeltaNet and Gated DeltaNet at every evaluated window size, and comes within 3.5 points of the full-KV attention reference at W = 32.

VNBench accuracy for IA-DeltaNet rises from 17.5% at 2 visible frames to 43.8% at 32, exceeding DeltaNet and Gated DeltaNet at each tested window size.
VNBench accuracy across LLM-visible windows. Dashed lines denote attention with all past keys and values.

Long-Document Language Modeling

We train on a 2K-token context and evaluate on PG-19 documents of up to 128K tokens, carrying recurrent states across consecutive chunks within each document.

DeltaNet and IA-DeltaNet start with similar perplexity, but diverge as the state is propagated. At 64–128K tokens, perplexity reaches 90.84 for DeltaNet and 25.51 for IA-DeltaNet.

Gated DeltaNet remains the most stable on this task. Forgetting and interference-aware writing address different parts of the update and can be combined.

PG-19 perplexity: DeltaNet rises sharply over long documents, IA-DeltaNet increases modestly, and Gated DeltaNet remains lowest.
PG-19 with document-level state carry. Lower perplexity is better.

Streaming Video Question Answering

We insert two memory blocks between the vision aligner and the LLM in Qwen3.5-4B. All fine-tuned models are trained with W = 8 visible frames. IA-DeltaNet achieves the best or tied-best accuracy among the evaluated fixed-size memories across these benchmarks and window sizes.

Streaming video QA accuracy (%) with the last 8 frames visible to the LLM.
ModelMemoryTempCompassNExT-QAMVBenchTemporalBench
Qwen3.5-4BNone66.372.758.758.0
+ IdentityNone64.971.955.964.2
+ DeltaNetO(1)66.473.558.364.6
+ Gated DeltaNetO(1)67.573.658.165.4
+ IA-DeltaNetO(1)70.176.959.265.4
+ Attention‡O(N)68.876.758.165.7

Memory cost is relative to stream length N. Identity removes the temporal mixer. Bold indicates the best result among fixed-size memories. ‡ Attention is the full-KV reference.

On Video-MME, IA-DeltaNet achieves 54.9%, compared with 53.5% for DeltaNet, 51.4% for Gated DeltaNet, and 58.4% for full-KV attention. This evaluation uses W = 8, 64, and 256 for short, medium, and long videos, respectively.

Understanding the Memory Writes

Direct measurements connect the improved retention to lower write interference. The diagonal approximation closely follows the exact solution when the covariance is diagonal; rotating the covariance exposes the limits of the approximation.

Under axis-aligned synthetic key covariance, diagonal and exact IA-DeltaNet have much lower interference than DeltaNet.
(a) Synthetic keys. With diagonal covariance, the diagonal approximation nearly matches the exact write direction.
With rotated synthetic key covariance, diagonal IA-DeltaNet lies between DeltaNet and exact IA-DeltaNet in write interference.
(b) Rotated synthetic keys. Off-diagonal structure creates a gap between the diagonal and exact variants.
IA-DeltaNet incurs lower write interference than DeltaNet on learned real-video keys throughout the stream.
(c) Real video keys. IA-DeltaNet (red) reduces interference compared with DeltaNet (blue), with a widening gap over the stream.
The median effective rank is 19 for DeltaNet keys, 49 for IA-DeltaNet keys, and 100 for IA-DeltaNet write directions, out of 160 dimensions.
(d) Effective rank. IA-DeltaNet write directions span a broader subspace: median rank 100 out of 160 dimensions, versus 49 for its keys.

BibTeX

The arXiv preprint and citation will be available here upon release.