How Lloyal turns the model's working memory into agents on the GPU

Ask an agent framework where an agent's memory lives and you get two answers. There is the checkpoint — messages, tool results, scratchpad — which lives in your application. And there is the KV cache, which lives in someone else's datacentre and dies when the response leaves.

Two memories, joined by serialisation. Almost everything that frustrates people about agent systems follows from that one fact.

What the industry calls an agent

Today an "agent" is typically a loop around a model call. LangGraph draws it as a graph: nodes are steps, edges are transitions, a checkpoint store holds a serialisable dict. AutoGen draws it as roles conversing. CrewAI as a crew with tasks. The vocabularies differ; the topology doesn't. The agent's state lives in the application, the model's state lives in the provider, and neither can see the other.

The consequence that matters most is that you cannot govern reasoning while it is happening. A client can inspect a finished output and decide whether to accept it. It cannot look at a trajectory halfway through and route evidence into it, narrow what it may conclude, or decide that this particular continuation should not become the thing the next step inherits. By the time your code runs, the reasoning is over. Every guardrail is a post-mortem.

The rest follows. Continuity is a transcript replayed into a prompt. Concurrency is N separate conversations with a stranger. And the bill multiplies per agent, because each agent is its own request.

The wrappers keep improving — better graphs, better retries, better graders — but the centre of gravity never moves. The agent stays a client of inference.

We spent a year on a different question.

What if agent state and model state were one and the same thing?

Not an agent that calls memory. An agent that is memory. If forking an agent were forking its state, what would fork mean? What would stopping mean? What would pressure mean?

The answers turn out to be KV-cache operations:

• Fork walks the shared cells and sets one more owner bit per cell. No allocation, no copy, no decode.
• Think is one tagged row in a batch, decoded alongside every other row in a single forward pass.
• Stop discards the branch and reconciles its cells back into the tenancy counter. Because hybrid models' recurrent state can never be rolled back, pruning isn't a fallback — it's the guard.
• Pressure is a counter the application reads and acts on. Not telemetry. An admission controller.
• Continuity is retention: keep the accepted lineage, release the working subtrees. What survives is a decision.

In that world an agent is a tenancy lease on live model state, and the application owns the lease. We call the architecture vertical inference: model execution becomes resident application state, and ordinary code programs topology, evidence, tools, policy, authority, completion and continuity while the intelligence is still alive.

The model supplies the intelligence. The harness supplies the institution.

A question isn't a product, so we built it, pointed it at the most expensive bill in AI, and published the traces.

One property, checked by someone else

Ten agents shared a single 239 GB model on Runpod. The whole session cost $16 — not per agent, total. GLM-5.2 at a 2-bit quant on two B200s for 82 minutes, published as reasoning.run with the cost breakdown, the exact config and each agent's annex.

Zhen Lu, CEO of Runpod — whose hardware we rented — wrote about it publicly:

"Multi-agent reasoning has a reputation for multiplying inference bills. This is the cleanest counterexample I've seen: ten agents thinking in parallel on frontier hardware, 82 minutes, sixteen dollars."

That validates one property: shared live state makes concurrency cheap. It is the easiest consequence of the architecture to measure and the least interesting thing about it. The claim that ordinary application code can govern that state is not something an invoice can settle, and the rest of this post is the part we have to earn.

Two trees, one system

The developer-facing view is written up in Thinking in Lloyal. Its opening line is the whole design:

A harness is a tree of owned lifetimes governing a tree of live inference state.

Structured ownership decides what lives and dies together. Application control decides what the owned execution does next. Those are two different questions, and most of the difficulty in agent systems comes from conflating them.

The left tree is ordinary TypeScript. withSpine opens a scope holding a live line of attention. agentPool spawns a cohort inside it. A tool opens a scope of its own. Every scope carries an ensure().

The right tree is KV cells. The spine's [system + tools] header is decoded once and prefix-shared, so the role and tool schemas occupy physical KV exactly once regardless of how many agents the pool spawns. Each agent's branch owns the cells above its own fork_head and shares everything below.

The two trees are the same tree at different altitudes:

TypeScript	Cells
spawn	seq_cp — add one owner bit per shared cell
a concurrent cohort	one batched decode, one row per branch
scope exit	prune — release cells above fork_head
halt	prune subtree, cascade, post-order

Orphaned branches are structurally impossible: every fork registers an ensure() that prunes on scope exit, and the pool carries one more.

Which gives the definition the codebase already uses: an agent is a branch with intent. The branch is the ground truth — the cells, the position, the sampler. The agent is the interpretation layer that gives that state meaning and decides what to do next.

Two details are where the design stops being a diagram.

The inheritance contract. A fork inherits state: the KV prefix, the spine header, the format config, an optional seed. It does not inherit judgment: policy, guards, grants and the tool registry are per-pool; tool history is read across the lineage when a guard asks for it, never carried. State inherits; judgment doesn't. Someone had to decide that, line by line. A wrapper has no such contract, because a wrapper has no children — only requests.

Three distinct deaths. Scope exit prunes. Wind-down drains: stop spawning, reap to recovery, let in-flight tools settle. Cancel discards one agent — discard, not drain: no recovery, prune, reclaim its cells for its siblings. Draining and discarding are different intentions, and the application says which one it means.

One tick, fiber to silicon

Walk a single commit tick downward.

The TypeScript pool fiber — single-threaded by design — samples each active branch from its logits_snapshot, a branch-owned copy, because the shared context's logit buffer is transient and valid only until the next decode. Branch independence isn't a metaphor. It's a memcpy of n_vocab floats.

BranchStore.commit flattens to exactly one token per branch and crosses the N-API boundary. The real work runs on a libuv pool thread, off the event loop, and the worker is accept-then-decode: clone each branch's sampler, grammar and metrics into RAII snapshots; accept the tokens; then the only realistic throw point, the batched decode. On a throw the sampling bookkeeping is swapped back — for perplexity accounting only. The KV and recurrent state are not restored. Not an oversight: gated-delta-net-style recurrent state mutates in place during decode and cannot be safely undone. Recovery is topological. Pruning is the guard.

Below that, liblloyal writes one row per branch — token, pos, seq_id, logits = 1 — and issues one llama_decode. In llama.cpp the batch is pre-split on token rows; the splitter never reads seq_id. One graph forward pass per ubatch. The attention mask is [n_kv, n_tokens], and the per-cell question — does this cell belong to this sequence? — is an O(1) bitset lookup.

Read that twice, because it contains the architecture: to llama.cpp there is no "N sequences" dimension. There are token rows, each tagged with a seq_id.

Agents are rows.

Ownership is enforced by the scheduler rather than by convention. Every native store operation issues from the tick loop's single fiber. Cross-fiber work — spawns, spine extends, fan-out tool completions, cancels — rendezvous through pending queues and drains on the loop. Spawns and extends batch into one store.prefill per tick; produced tokens are one store.commit.

Pressure is an admission controller

The counter that makes concurrency safe is the same one that makes it programmable. remaining = n_ctx − cells_used_, partitioned into three zones: headroom, where new work is admitted; a soft floor, where the pool stops spawning and oversized results defer; a hard floor, where work is stopped before the next decode.

The hard floor is validated at startup against the native batch size — hardLimit ≥ nBatch — and the pool refuses to boot otherwise. Spawn admission reserves for batch-mates. Settlement admits what fits and defers the rest. Recovery is budgeted so the whole cohort's final reports fit inside one batched tick.

That last one is the part we're fondest of: the institution plans its own funerals within budget. An agent pool that can be asked to wind down has to know, in advance, that it can afford to hear from everyone before it dies.

This is what "programming surface, not telemetry" means concretely. The pressure counter isn't something you graph after the fact. It's the input to admission decisions your policy code makes, before the decode that would have violated them.

The cost model

Three laws, verified against source:

1. Dispatch count is O(1) in branch count. N branches whose rows fit in n_ubatch are one forward pass. commit is the degenerate case — one token per branch — so ten agents are ten rows are one dispatch.
2. Per-tick wall-time is O(n_kv × rows). Adding a branch adds one query row and one O(1) bitset test per cell. There is no × n_seqs multiplier anywhere. Branch count is nearly free in time; KV fullness is the cost.
3. Concurrency is paid in space, and parked concurrency is free. cells_used_ grows with live decode and settled results. An agent parked awaiting a tool is skipped at zero cost — no turns, no tokens, no cells.

Setup obeys the same law: the shared header is decoded once and inherited by every fork. You pay for the cache, not the agent count, at setup time as well as decode time. That is the $16, mechanically.

And the failure mode, because a post that hides them is marketing. We call it the 519-second decode, after the trace that caught it. A ~1,400-token recovery report over a near-full ~25–30k-cell cache is slow, because every tick is one llama_decode over a large n_kv. Two concurrent reports and ten decode at the same per-tick speed — the cost is the cache, not the cohort. The cure is to reap earlier, which is orthogonal to concurrency and is exactly what the pressure zones exist to do.

Why this doesn't reduce to an API call

Two objections are worth answering directly.

"Prefix caching already does this." It doesn't. Prefix caching deduplicates: two requests that happen to share a leading token sequence reuse the same blocks. That's a content-addressed hit, decided by the server, at a request boundary. It is not a fork. You cannot say branch here, at this position, now; hold four lineages over one trunk and decode them in the same step; fan them back in; or run a DAG whose nodes are KV sequences and whose edges are what each node inherits. A cache hit buys you cheaper prefill. A branch is an addressable object with a position, a parent and a lifetime — one your code owns, prunes, and decides the inheritance of.

"Then a serving engine could expose it." We are the serving engine. Native targets run in-process; web targets run against lloyal-host. That isn't a technicality — these are operations on serving internals, so whoever exposes them is the engine. There's no version of this that ships as a feature on someone else's endpoint, because an endpoint's unit of sale is the request, and the request is the object this dissolves.

And nothing on the sampling side crosses a wire in either direction. A branch owns a mutable copy of the logit distribution, a cloned sampler chain carrying its own penalty state, and its own grammar parse state that diverges from its sibling's. Merging two branches' distributions is an in-place write into one of them. There is no request shape that hands a client the distribution and takes it back.

"It's a wrapper." A wrapper calls a server. We are the server. The trace is the appendix.

Owning the execution instead of optimising the request is what the whole system is for. It's also why the interesting part isn't the invoice.

Run it

The same harness that ran the $16 datacentre session runs a 4B model on a laptop. Identical code, different silicon.

npx harness.dev@latest new

Node 24+, about a minute of installs, and a ~2.7 GB model fetched and digest-verified on first run. No API key — the model lives inside your app.

Pick the basic template and run npm run dev:web. You get a working research harness with two agents reading in parallel over a shared spine.

That number in the header is the pressure counter from three sections ago, running on your machine. It's the same remaining = n_ctx − cells_used_ your policy code reads.

Then make one edit. Find the orchestrator and swap parallel for chain. Re-run.

The agents now fork sequentially from a spine that each one extends before the next inherits it — same code, same model, different topology of live reasoning, because you changed one word in an application file. That's the whole idea in a single diff, and it takes about ten seconds.

Build your first harness walks the rest.

---