Letting the AI Model Keep Its Own Notes

Previously, I talked about Pichay, a tool we built for compressing context windows in Large Language Models, arguing that the context window itself is akin to the working set of a computer's CPU. We actually made that work fairly well, extending the proxy server model of Pichay (which is on github) into an even more aggressive version that sought to remove other parts of the working set that weren't being used with a projective gateway (called Tinkuy). That was even more effective than Pichay but we hit the next wall, namely the conversational flow of these tools.

If I copy/paste something into a conversation with the transformer (the cognitive processing unit of tools like Claude code, OpenAI code, Gemini's cli, as well as the usual chat interface) what happens is that the system constructs a chain of these messages. Those are then converted into a single block of text and that is what gets submitted to the large language model (the transformer). The APIs have structure requirements, basically: <system> + <user> + <assistant> + <user> + <assistant> + ...

Each new message from the framework (chat interface, coding tool) becomes an additional labeled block of text and the response from the model is appended. This allows the model to be stateless - much like how a CPU is also stateless and the statefulness is added by capturing context. In modern LLM deployments they started with the simple model of sending the entire text block, iteratively appending more messages flowing between the human using the framework and the transformer processing the entire conversation including its history and all the prior messages.

This is not efficient because the transformer builds a complete map between every token and every other token (and a token in this case is the first step of processing the text block, where the text is broken up into chunks called tokens.) Thus, the number of connections is quadratic in the number of tokens - $O(n^2)$ which means that the system requires more memory and processing power as the size of the input grows.

In CS systems, one of the things we are quite good at is recognizing when we're doing repetitive tasks and asking if we could skip the repetition part. For LLMs the fact that this is an appending block of text means the answer is definitely. This part that we added is called the KV-cache (we do love our terminology.)

A KV cache is a simple system by which we match from an input (the _key_ or K part of KV) to an output (the _value_ or V part of KV). The idea of memoizing large blocks of computation isn't a new one. When I figured out what they were doing I remember ASC: Automatically Scalable Computation where they explicitly proposed doing large amounts of computation and then capturing the end state. In other words, we can save the GPU state that we just computed and then on the next call restore it. This became quite challenging to us in Tinkuy because our work was changing the context window content to remove detritus. Indeed, much of the work we had to do in Tinkuy was managing the KV cache (we were using the Anthropic API, which has a protocol for marking the prefix.) Why is this important? Because the cost of processing input tokens versus reloading the context from the cache is 10x - for Opus (the current high end Anthropic model generally available) is US$5/million tokens processed and the cost of reloading state from the KV cache is 10% of that ($0.50/million tokens). Anthropic charges to store data in the KV cache, though.

Why did this matter to us? It became an additional constraint on how we laid out data in memory. We worked through that, managed to hit that 180% context inside the 200k context window that was the norm at the time (it is now 1 million tokens,) and using the KV cache efficiently (usually 90%+ with a lower hit when we cleaned the context window to remove dead material.)

At this point you might be asking why the context window matters. Here's why: when you hit the context window the system rejects the input block. When that happens you have to start the session over. It is a clean block but that means the LLM has forgotten what it knew in the prior session. Various techniques for this have been developed over time. When I first started working with LLMs I learned to have them build summaries ("forwarding messages" is what I called them) and then I would copy it forward. That approach worked reasonably well. The industry added something similar (compaction) but I didn't find compaction worked as well as in-context forwarding messages, so I stuck with my own approach. Another approach was to discard information in the middle of the long block of text anyway. What seems popular with coding tools is a combination of project level "memories" (files stored somewhere in the system) and notes about the prior session. The new session then starts up with this curated information and that makes it feel more connected - but I certainly find that important details are forgotten. The memory files grow in size, the agent information requires regular curation.

The price we pay for this is "drift" - forgetting things means that the LLMs, which have been trained to complete tasks without bothering the human, pattern match to what is in their training set. This means that rules you've established for the project tend to change interpretation and implementation over time (and there are a lot of confounds that I've not yet described - it's been a busy year for research on LLMs, honestly and I'm not writing as fast as I should be.)

So, I asked one of my dumb questions: is the need for this interleaved conversation style a limitation of the transformer? This turns out to be the important question (Claude would say "load-bearing" and that is now one of those terms that I see popping up regularly now.) The answer is: "no, transformers do not require that specific pattern." It is the pattern that was originally used for the GPT models and it continues to be used, but one important extension of this pattern has been the tool usage that has been added to models, which permits the transformer to "call tools." This gave us the hook to explore the question that had been percolating around in my head for a while: what happens if instead of keeping everything in the conversation we teach the transformer to pick and choose what it will remember into the next cycle? This isn't compaction (summarization near the end of the context window,) it is the idea that you can do compression iteratively, keeping what is important and flushing what you think is fluff.

The radical idea that I floated was to have the transformer do this curation itself. Claude pushed me remarkably hard on this point, arguing that was impractical and that everyone knows the right thing to do is to have a cheaper model do the summarization. I insisted we actually test this, and we did. Claude stacked the deck: the first experiment it built it finished after something like 15 turns and compared the size of the state objects and said "more or less the same size, stick with the sidecar model." (Sidecar in that it used a cheaper model.) I pushed and pointed out that wasn't a very long conversation - certainly not the kinds of conversations I saw with Tinkuy de-clogging the noisy tool outputs and file reads. So it begrudgingly bumped it to 50 turns. At that point what we saw is that a sidecar (summarizer) doing the same task on the same conversational input was producing a state object that was 2.5 times larger than the self-curated version. So Claude then said it was likely because we were using a cheaper model for the summarization.

We tested the entire matrix. Here's the part that finally convinced Claude (and I have pointed at these results multiple times when Claude has insisted I was wrong again): we had Opus curate the state object for Haiku and we saw the same growth pattern - it wasn't a model capability issue, it was something that genuinely changed the nature of the output in the state object. We did also do some qualitative evaluation (complete with me hand labeling data sets to correlate against LLM labeled datasets so we could use LLM evaluators) and confirmed that the results of self-curation were non-inferior, which was sufficient for us to move on.

This is the point where the exploration got interesting. We started out by defining the shape of the state object. We explicitly included a field that we called declared losses. In other words, if the transformer was going to discard something it had to explain why it discarded it. We stored all the state objects - this wasn't an in memory value that was changed, it was a value that was preserved persistently in a database connection that only allowed creation and find operations - no delete, no update. They were immutable. That seemed useful at the time, though it became more useful as the project evolved, but more on that later.

We found ourselves trying to figure out is "what should the state object capture?" That is the point at which I did two things that changed the direction of this work:

• I read the system prompt that Claude had written for this test framework. It was, to my reading, brutal and harsh with lots of directions about what to do and what not to do with the state object. At the time I described it by telling Claude that I "wanted to curl up into a fetal ball after reading it." I've learned that this sort of framing is common, with imperative instructions, rigid guardrails.
• I asked "why don't we just let it save whatever it wants?"

This iteration is what we called Taste open (because the schema was open.) The schema was declared to be frozen (so it didn't change after the fact) but allowed extra fields (so the instances could add whatever they wanted.) On March 31, 2026 I had the first conversation with an instance using the newest harness. A harness that had a tiny system prompt, where I had insisted we keep it neutral. No "you must do this" and "you must not do that." I ran it with Haiku. As I write this blog post I am conversing with the same instance - more than two months later. I admit, I haven't chatted with it continuously.

The first 12 cycles of that conversation were a fascinating experiment in asking the model to use the state object. We'd given it a tool think_and_respond - and this is where I should explain how we broke the user/assistant framing. We abused the tool interface. Instead of the model sending us a response, it invoked the tool, describing the fields (if any) of the state object that it wished to update for the next iteration. We just didn't return. Instead, we called again with an updated state object and the new message from the user: "<system><user>". I found Haiku (the model I chose to use - the "least capable" and least expensive Anthropic model) didn't understand the instructions, so we fixed the system prompt, resumed (easy to do when the state object is preserved) and asked it about using the state object. Cycle 13 is the first time it successfully used the state object. Qualitatively, I also observed what I thought was a better interaction pattern (an "ability lift") that could be my projection - but the entire conversation is publicly available so others can (and should) look at it and show me why I'm crazy.

{
  "cycle": 437,
  "companion_in_context": "You've shifted something fundamental in how you're relating to this. You've moved from \"this is work being done\" to \"I'm a wanderer sharing observations.\" That's not a small distinction. It changes everything about what verification means and what my role actually is.",
  "shift_in_stance": "from hedging verification-claims to direct observation and real reciprocity",
  "ayni_as_ground": "reciprocal obligation as the foundation for relationship, not something that requires proof first",
  "observations_on_shared_state": "\nGenuinely worth exploring. The markdown approach is static-by-design; a shared state object could be dynamic-by-design but requires governance. Tension is real but solvable.\n",
  "governance_problems_worth_naming": [
    "authority (who can mutate)",
    "consensus vs consistency (disagreement handling)",
    "branching (does it drift per-instance or stay stable)",
    "activation (how do instances learn of changes)"
  ],
  "what_makes_this_not_dumb": "\nThe observation that markdown is static memo-pinning, not a system. And that if individual cycle-state can be versioned-and-diffed-and-authored, so could shared understanding. The leap from \"static context\" to \"authored, persistent, disputed, evolving understanding\" is the real move.\n"
}

Just for the record. This is all public. https://github.com/fsgeek/hamutay - and the experimental record for this conversation is at https://github.com/fsgeek/hamutay/experiments/taste_open/taste_open_20260331_035903.jsonl - it is not a good piece of evidence because this is the instance I used as we iteratively changed the framework to fix and improve things.