Kernel Adaptive Memory: testing persistent kernel memory for sequence models

Can a sequence model keep the useful parts of attention while giving persistent memory an explicit kernel geometry and a fast path to adaptation? After six experiment phases the honest answer is: not in the general form I first proposed, but a narrower version of the idea is still open.

The question

Kernel adaptive filters such as kernel least-mean-squares (KLMS) predict with a bank of localized basis functions centered on stored samples, and they adapt online with cheap coefficient updates. Transformers learn context-dependent query/key routing and are far more general, but their memory is a growing context window of pairwise comparisons. KAM asked whether the two could be combined: give a sequence model a finite bank of learned persistent supports with an explicit kernel geometry, and keep a fast linear path for adaptation after the data changes.

The July 2026 working note framed this as a falsifiable program rather than a result. It defined matched baselines, the diagnostics that would be reported, and the conditions under which the idea should be declared not worth pursuing. Most of what follows is the record of running that program and letting it narrow the claim.

Phase I: one kernel calculation, two memories

The first implementation replaced scaled dot-product attention with a radial kernel score: similarity is a distance in a learned feature space under a positive diagonal metric, rather than a dot product. The same operator is used in two roles. In context attention a token queries earlier tokens from its own sequence. In persistent-memory attention a token queries a finite learned bank of M support keys and values that is shared across all sequences.

Because one primitive serves both roles, the model can be ablated cleanly: drop the support bank, drop context attention, or swap the radial score for the familiar dot product while keeping everything else fixed. Seven command-line variants (the full KAM, kernel-only context, memory-only, a dot-product transformer, a dot-product hybrid with supports, a GRU, and an MLP) and four tasks make up the original grid.

Context attention

Causal retrieval from the current sequence with a radial score; an optional local window W tests whether bounded context is enough.

Persistent supports

A learned bank of M key/value pairs that any sequence can query, intended to hold reusable structure rather than another copy of the context.

Adaptive readout

After a Mackey–Glass delay shift, the feature extractor is frozen and only the scalar readout is updated with normalized least-mean-squares (NLMS).

What the first matched grid showed

The first grid was 45 runs: five attention variants, three tasks (copy, hidden-regime grammar, Mackey–Glass), three seeds. It did not show a general advantage for the kernel geometry. Dot-product attention won or tied on copy and regime grammar. On Mackey–Glass the full KAM beat the dot-product transformer, but the dot-product hybrid, which adds the same support bank to ordinary attention, beat KAM; the useful ingredient there was the memory bank, not the radial score. On Tiny Shakespeare the dot-product transformer reached lower perplexity (6.96 versus 7.41) in about half the runtime, and a KAM model trained on copy length 16 fell to 16.1 percent accuracy at length 32. KAM also carried roughly 1.3 times the parameters, so the comparison was not even parameter-matched.

One signal survived. After a Mackey–Glass delay shift, adapting only the scalar readout with NLMS cut post-shift mean squared error by 44.5 percent relative to the frozen readout, and by 55.8 percent late in the stream. That is the KLMS-style property the project set out to keep, and it pointed the follow-up toward memory for nonstationary streams rather than toward a new attention score.

Phases II through V then hardened the method more than the architecture. A five-seed paired grid (50 runs) found no Holm-corrected comparison whose confidence interval excluded zero. A 1,728-row development search on switching dynamics moved the program onto HiPerGator. A bounded data-regime screen was filed as diagnostic only, and a learned-versus-fixed-feature campaign introduced active-parameter matching to within 1 percent and machine-checked validity gates that must pass before any result is read.

Narrowing to sparse separable memory

The Phase VI brief opens by stating that the broad claim, that the dense KAM bank should replace transformer attention, is not supported. The remaining hypothesis is narrower: a transformer may benefit from a sparse, support-derived memory whose values can be solved or adapted quickly while its support geometry changes slowly. Attention keeps modeling context; the memory branch supplies selectively accessed local transformations.

Concretely, each decoder block maps its residual stream to a routing coordinate, scores all M support keys with a radial kernel distance, and keeps only the top K (with K much smaller than M). Each selected support owns a locally affine expert, and the routing-weighted sum of those experts re-enters the residual stream through a zero-initialized gate, so an untrained memory branch reproduces the baseline model exactly. Stored capacity scales with M; active expert compute scales with K. Routing cost is a separate question, which is why the router zoo includes exact, chunked, product-key, and approximate variants with recall measured against an exact reference.

"Separable" refers to two parameter classes on two timescales. The fast algebra (expert values) is linear given the routing and can be solved by ridge regression or streaming recursive least squares; the slow geometry (keys, query map, metric) is updated by gradient steps, alternated with the algebra at ratios such as 8:1 or 32:1, and in the registered lifecycle frozen for the last 20 percent of training so the final tuning happens against stable routing.

T-KAM-F

Fixed or data-centered keys: is useful geometry available without learning key positions at all?

T-KAM-L

Learned keys, optionally as a low-rank delta from initialization: does learned support placement help?

T-KAM-ALT

Alternating algebra/geometry updates: does timescale separation improve optimization?

Controls

A plain decoder (T0), a widened decoder (T-WIDE), learned memory tokens, mixture-of-experts, and product-key memory at matched total or active parameters.

The falsification discipline

Running a single-person architecture idea for six phases invites wishful reading, so the process is built to resist it. From Phase IV on, every phase has an implementation brief, a handoff document that asks for critique of the next step, a decision memo with outcome-to-action rules written before results are inspected, and a reproducibility record; earlier phases have a subset of these. Campaigns are immutable JSONL manifests where one row is one independently rerunnable job, submitted as Slurm arrays on HiPerGator's NVIDIA L4 nodes, with aggregation and report jobs held behind dependency gates.

The training seed is the inferential unit. Comparisons are paired by exact seed identity and reported with bootstrap confidence intervals, sign-flip randomization tests, Holm correction within declared families, and two-one-sided equivalence tests with declared margins (±1 percent in the current Stage 1 decision memo), because a non-significant difference is not evidence of equivalence. Confirmation campaigns fix their sample size in advance from a planning effect deliberately smaller than the pilot effect, to protect against winner's curse, and forbid optional stopping or seed replacement.

Independent audits catch what the pipeline gets wrong, and they have. Phase VI Stage 1 was rerun after audits found a geometry-factor coverage gap and a task/optimizer dispatch defect; an overnight campaign's adaptation rows were reclassified when an audit showed the declared adapters were never dispatched; and the Phase 6.2 mechanism study, 168 rows at 50 million tokens each, completed training and then failed its own preregistered validity gate because CUDA top-K tie-breaking was not invariant to matched key/expert permutations. That defect was localized to late memory layers, reproduced on three L4 hosts and absent on CPU, fixed with an opt-in support-identity tie-breaker, checked with a locked 26-checkpoint replay and a 24-row pilot, and a fresh 168-row rerun was preregistered rather than patching the original campaign.

Where it stands

As of 2026-08-09 (the last commit), the repository holds the Phase I reference implementation, the Phase VI decoder, memory, and optimization packages, and the campaign infrastructure, with 109 tests across all phases. On HiPerGator, Phase VI's Stage 0 correctness gate passed 128/128 rows, the corrected Stage 1 mechanism screen completed 3,000/3,000 rows with passing audits, and bounded Stage 2 through 6 profiles completed and were audited as implementation and resource evidence only. In the Stage 3 router profile, exact, chunked, and product-key routers recovered the exact reference's top-K choices with recall of 0.993, 0.991, and 0.983; the approximate router managed 0.477.

The quality evidence is thin and honestly labeled. A three-seed overnight language-modeling pilot put the fixed-key variant's validation loss 30.4 percent below a widened transformer at the registered token checkpoint, but with n = 3 the paired permutation p-value was 0.25, and the corrected decision was to retain the result as diagnostic only. The 156-row confirmation that followed (30 TinyStories pairs and 24 Tiny Shakespeare pairs at 50 million tokens per row) is referred to in later planning documents as not having met its primary superiority gate; its full report is not in the repository, so I do not quote its numbers here. The Phase VI decision memo's standing verdict is that no promotion decision has been made.

The current thread is a mechanism study of what sparse memory actually learns, not another attempt to win a benchmark: whether learned keys beat fixed ones, whether freezing geometry late helps, and whether alternating updates matter, with checkpoint-reuse causal checks (support and layer deletion, permutation tests) required before any further scaling. The preregistered support-identity rerun is the next result; whether it launched and what it found is not recorded in the repository as of this writing.