How Halo remembers you

Most AI assistants do not really build a picture of you. They save your memories as little scraps of text, one after another, with nothing connecting one to the next. Ask one how your dentist relates to your insurance claim and it cannot say, because it never drew the line between the two in the first place.

Real memory does not work like that. Your brain does not keep a flat list. It stores things as a web of neurons that wire together and form connections, and a lot of the meaning lives in those connections, not just in the individual facts. Halo’s memory works the same way. It builds a connected graph of your life, where every fact links to the people, places and events around it. The difference is that this graph is completely private and generated locally, on your own device.

The approach is loosely based on Andrej Karpathy’s idea of an LLM wiki: a personal knowledge base kept as linked pages that a language model writes, maintains, and reads back. Halo takes that idea and makes it self-building and private.

This post is a walk through how that actually works, kept as honest and concrete as I can make it, because the design choices are the interesting part. If you have ever wondered what it would take for an assistant to genuinely understand your life without shipping that life off to a data center, this is my answer.

What I actually wanted

Tools like ChatGPT have a memory feature now, and it is genuinely useful. But under the hood it is mostly a growing list of short statements the model chose to keep, stored in the cloud, with little structure tying one fact to another. It can recite what it saved. It cannot reason across those facts, and it cannot notice that something it is about to rely on stopped being true months ago.

I wanted three things that a flat list does not give you. First, memory should have structure, so the assistant can follow how people, places and events connect. Second, it should be honest about time, because some facts are permanent and some go stale within days. Third, and this was non-negotiable, it should live on your device, not on mine.

Read it, search it, traverse it

The simplest way to understand Halo’s memory is by what you and the assistant can actually do with it. There are three things: you can read it, you can search it, and you can traverse it.

Underneath, those three capabilities sit on just two stores. The first is a set of structured Markdown pages: one page per real thing in your life, a person, an employer, a trip, a recurring bill, linked to each other with wiki-style links so the whole thing becomes a graph you could open and read. The second is a vector index, built by splitting those same pages into small chunks and turning each chunk into a vector on your device, so Halo can find things by meaning rather than by exact words.

Notice that two of the three capabilities, reading and traversing, act on the very same Markdown pages. A person reads them as notes; the agent walks them from the terminal. Only searching uses the second store, the vector index. So it is three ways to use your memory over two places it actually lives.

Keeping those two stores separate matters. The Markdown graph is the source of truth that a person can inspect and correct. The vector index is a derived, disposable view that exists purely to make retrieval fast. If the index were ever lost, it could be rebuilt from the pages. That is a much healthier relationship than storing your memories only as opaque numbers in a database.

How a memory gets made

New information shows up constantly: a calendar event, an email, a transaction, something you did in the app. Halo does not call a model every time a byte changes. Instead it runs a careful loop that does the least work necessary, and only reaches for the language model when there is genuinely something new to understand.

A few details here are worth calling out, because they are what keep this from being slow or expensive. Halo hashes every source and skips anything that has not changed, so unchanged data never reaches the model. For sources that only ever grow, like an event log, it sends only the new lines rather than the whole file. And before the agent writes anything, it is handed a deterministic map of every page that already exists, built in plain code with no model involved.

One page per person, not a pile of events

The biggest decision in the whole system is that memory is entity-centric, not event-centric. A naive memory stores one record per event and quickly drowns in thousands of tiny, disconnected entries. Halo does the opposite. It keeps one page per real entity and records dated entries inside that page. Your colleague Sarah has a single page that accumulates context over time, not forty rows scattered across a table.

memory/
  ├── _index.md        catalogue of every page
  ├── _log.md          chronological ingestion log
  ├── USER.md          you, the centre of the graph
  ├── People/
  │     ├── Family/
  │     ├── Friends/
  │     └── Professional/
  ├── Work/
  ├── Health/
  ├── Travel/
  ├── Finances/
  ├── Events/         trips, weddings, conferences
  └── Misc/

Each page is plain Markdown with a small block of structured fields at the top. That structure is what lets the rest of the system stay deterministic, and it is also what makes a page readable on its own. Here is roughly what one looks like.

---
title: "Sarah Chen"
aliases: [Sarah, [email protected]]
email: "[email protected]"
created: 2026-05-24
updated: 2026-06-20
staleness: 2          # 2 to 4 means it changes over years
tags: [colleague, acme]
---

Met at the Acme offsite. Leads the design team and
prefers async updates over calls.

## Related
- [[Work/Acme]] (employer)
- [[People/Professional/David-Kim]] (manager)

One-off but genuinely significant things still earn their own page. A wedding or a big trip is worth remembering as an entity. The filter is lasting significance, not whether something repeats. The goal is a memory that stays dense and coherent rather than one that grows for the sake of growing.

Links that carry meaning

The links between pages are not just navigation. Each relationship is written on both pages, and each side carries a short label from its own point of view. Sarah’s page links to Acme as her employer, and Acme’s page links back to Sarah as an employee. That two-way labeling is what makes the graph useful in both directions when the assistant is reasoning.

There is also a rule I care about a lot: a link has to be earned. Two people showing up in the same batch of data is not evidence that they are connected. A relationship only gets written when there is actual evidence for it in a source or an existing page. This is a deliberate guard against the thing that makes AI memory feel untrustworthy, which is confidently inventing connections that were never real.

Knowing when a fact has gone stale

This is my favorite part, because it is the thing most memory systems ignore. Not all facts age the same way. Your date of birth is true forever. Your job title is true for a few years. Whether you are still waiting on a reply from someone is true for maybe a day. Treating all of these as equally current is how an assistant ends up confidently telling you something that stopped being true months ago.

So every page gets a staleness score from 0 to 10 when it is written, judged by the nature of the information rather than the clock. Crucially, it is set once and never recomputed, so nothing has to sweep through every page to age it. Paired with the date the page was last updated, that single number lets retrieval decide whether to trust a fact as-is or go re-verify it against a live source.

The payoff is subtle but important. A name with a staleness of 0 is trusted no matter how old it is. A high-staleness page that has not been touched recently is treated as a lead to verify, not a fact to state. That is much closer to how a careful person handles their own memory.

Finding the right memory at the right moment

When Halo needs to recall something, pure vector search on its own is not enough. Searching by meaning is great for natural questions, but it can fumble exact strings like an account number or a specific name. So Halo takes a hybrid approach and blends two signals: most of the weight goes to semantic similarity from the vectors, and a smaller share goes to plain keyword overlap to rescue the exact-match cases.

Results come back chunk-first. Each result tells the assistant which page and line range it came from, along with that page’s update date and staleness. The assistant opens the full page only when it actually needs more, and follows the typed links from there. It is a bit like getting search snippets first and clicking through only when a result looks right, which keeps the reasoning focused instead of dumping everything into context.

But search is only half of how Halo recalls things. The other half is that the memory is just files on disk, and the agent has a terminal. So alongside the vector search, the agent can walk the memory graph directly, the way you would explore a project in a code editor. It uses ordinary tools like find and grep over the index and the pages, lists a category to see what is there, opens a page, and then follows the typed links from one entity to the next until it has gathered exactly the context the question needs.

These are two retrieval paths over the two stores from earlier. Search runs over the vector index; traversal runs over the Markdown graph itself, the same files you could open and read. They cover each other’s blind spots. Vector search is great when you describe something loosely and cannot name it. Graph traversal is great when recall depends on connections, like pulling everyone tied to a particular trip, or following a person to their employer to a shared project. One finds things by meaning, the other by structure.

Where all of this lives

Here is the part that ties back to why I built Halo in the first place. The memory pages live on your device. Halo ships with its own embedding model, so turning those pages into vectors also happens on your device, with nothing sent out to be processed. The vector index is stored on your device and, if you use iCloud, synced privately across your own devices through your own iCloud account. None of it passes through a Halo server, because there is no Halo server.

I want to be precise about one thing, because being vague here would be dishonest. The language model itself is not on-device. When Halo needs to reason, it sends a request straight from your device to the AI provider you chose, using your own key or subscription. So prompt content does leave your device for generation, but it goes to your provider, not to us, and your memory graph and vectors stay with you.

And as local models keep getting more capable, depending on a server model will increasingly be a thing of the past. If you want zero data leaving your control today, you already can: host a model with more than four billion parameters on a desktop with Ollama, use Tailscale to tunnel into it privately, and point Halo at that Ollama server. At that point even generation happens on hardware you own.

Why I think this is the right shape

A readable, self-updating graph that lives with you, knows how facts relate, and is honest about when they expire is harder to build. But it is the version I would actually want pointed at my own life.

That is the bet behind Halo. You should not have to hand over control of your data to get the benefits of an assistant that truly knows you. If that is the kind of assistant you have been waiting for, I would love for you to try it.