Fractal Memory Architecture: How We Gave Our AI Agent a Permanent Memory

Layer 1: Daily Notes (memory/YYYY-MM-DD.md)

This is the raw feed. Every session, Larry appends to today's daily note. Not a sanitized summary — the real stuff. What we built, what broke, what decisions we made and why, which files we touched, what commands we ran, what errors we hit.

Think of it like a ship's log. Not written for posterity — written to capture what actually happened while it's fresh. The detail here is high because nothing has been thrown away yet.

At session start, Larry reads today's daily (if it exists) and yesterday's. That's it for the daily layer — just the last two days at full fidelity.

Layer 2: Refined Dailies (memory/refined/YYYY-MM-DD.md)

Every night at 11:30 PM, a cron job fires. It spawns a Gemini Flash sub-agent, hands it the raw daily notes plus any session transcripts from the day, and says: distill this.

The output is a structured summary with clear sections: decisions made, things built, problems encountered, blockers, and next steps. It's maybe 20% the length of the raw daily — but contains 80% of the important signal.

These refined dailies are what the weekly rollup reads. They're also what gets loaded if you need context from earlier this week — you get the essence without wading through raw logs.

Layer 3: Weekly Summaries (memory/weekly/YYYY-Wnn.md)

Every Sunday at 11:45 PM, another sub-agent reads all seven refined dailies from the past week and synthesizes them into a weekly summary. The focus shifts from events to patterns: what were the dominant themes? What's the project trajectory? What worked, what didn't?

This is the layer that gives context like: "Week 3 was mostly debugging the Telegram bot — we got it stable by Thursday, then pivoted to the credential vault." That's the kind of trajectory context that makes an agent feel coherent across days instead of just day-to-day.

At session start, Larry loads this week's weekly summary. One file. Full week of context.

Layer 4: Monthly Overviews (memory/monthly/YYYY-MM.md)

On the first of each month at 12:15 AM, a sub-agent reads all the weekly summaries from the prior month and produces a monthly overview. The zoom level here is strategic — what was the overall arc? What are the biggest lessons learned? What changed about how we work?

The monthly rollup also promotes key lessons up to long-term memory. If a pattern emerged strongly enough to show up across multiple weeks, it probably belongs in MEMORY.md permanently.

Layer 5: Long-Term Memory (MEMORY.md)

This is the curated distillation. Not a log. Not a summary. A set of things that are permanently true and worth remembering — the kind of knowledge a human would hold in actual long-term memory rather than their notes.

Things like: architectural decisions we're committed to. Preferences. Hard lessons from mistakes we don't want to repeat. Recurring patterns about how we work. Context about ongoing relationships and projects.

Larry reads MEMORY.md at the start of every main session. It's the most important file he has — but also the most compact. Because it only contains what's genuinely important enough to carry permanently.

Layer 6: Sticky Notes (memory/sticky-notes/)

A folder of markdown files for persistent facts that don't change much: devices (what hardware exists and what it's for), accounts (usernames, platforms, chat IDs — no secrets), preferences (how we like things done), infrastructure (servers, SSH details, deployment configs).

These get loaded every session, always. They're small and static. The agent updates them whenever something new is learned — like "there's a new server" or "we switched from provider X to provider Y."

Agents can read and write them natively

An LLM reads and writes text. Markdown files are text. The agent can read its own memory files directly — no API call, no query language, no ORM, no schema to manage. Writing a memory update is literally just: write these lines to this file. That's it. No database connection, no transaction, no index to update.

You can read them too

Open any memory file in a text editor and you can see exactly what your agent knows. Want to audit what Larry remembers about a project? Open memory/2026-02-15.md. Want to see what he learned last week? Open memory/weekly/2026-W07.md. No database client required. No schema to decode.

This transparency matters. When something goes wrong — and it will — you want to be able to inspect the agent's memory directly. With a database, you have to write queries. With markdown files, you just read.

Git-trackable history

Every change to the memory files is a diff. Commit your clawd folder to git and you have a complete history of how your agent's memory evolved over time. When did it learn a particular fact? What did it know before a specific decision? You can answer those questions. With a database, you'd need audit tables and change logs — extra work that most people don't build.

Editable by humans, not just agents

Sometimes you need to correct something the agent knows. Maybe it learned something wrong. Maybe circumstances changed. With markdown files, you just open the file and edit it. Delete the wrong line, add the right information. The agent will pick it up next session.

With a database, you're running SQL queries or building an admin UI. That's overhead that slows everything down and rarely happens at all.

No infrastructure to maintain

A database is a service. It needs to be running, backed up, and accessible. If it goes down, your agent loses its memory. Markdown files are just files on disk. They're always there. They don't go down. They don't need connection strings or credentials.

Token efficiency through compression

Day 1 and Day 100, the session boot loads roughly the same number of tokens. That's the magic. Raw logs accumulate indefinitely, but the agent doesn't load raw logs — it loads summaries that are proportionally sized to their time horizon. More time = more compression = same token footprint.

Nothing gets lost

The raw daily files preserve everything. If you ever need to go back and look at the full detail of what happened on a specific day, it's there. The summaries are lossy by design — but the source material is always available.

Self-maintaining

The rollups happen automatically. The cron jobs fire, the sub-agents run, the summaries get written. You don't manage the memory system — it manages itself. The only human involvement is deciding what goes into MEMORY.md permanently (though the monthly agent handles that too).

Compaction-resistant

When a session's context gets compacted because we hit token limits, the session memory disappears — but the file system doesn't. The daily notes were written throughout the day. When Larry starts the next session, he reads the files. The compaction never happened, from his perspective.

Separation of concerns

The main agent doesn't run the rollups. Sub-agents handle that. The main agent's job is to do work and write to daily notes. A separate nightly process handles the rest. This keeps the main agent focused and the memory system running on its own schedule.

A file structure

clawd/
  memory/
    2026-02-17.md          # Today's raw daily
    2026-02-16.md          # Yesterday
    refined/
      2026-02-16.md        # Yesterday's refined daily
    weekly/
      2026-W07.md          # This week's summary
    monthly/
      2026-02.md           # This month's overview
    sticky-notes/
      devices.md
      infrastructure.md
      preferences.md
  MEMORY.md                # Long-term curated memory

Instructions in AGENTS.md

Tell your agent explicitly: every session, read today's daily, yesterday's daily, this week's summary, MEMORY.md, and sticky notes. In that order. Before doing anything else. Make it a rule in the agent's governing instructions.

Cron jobs for rollups

Set up three cron jobs: one at 11:30 PM nightly for daily refinement, one Sunday at 11:45 PM for weekly rollup, one on the first of each month for monthly overview. Each spawns a sub-agent (we use Gemini Flash for cost efficiency) that reads the source files and writes the summary.

Prompts that produce good summaries

The quality of the rollups depends heavily on the prompts. We found that structured output works better than freeform — ask for specific sections (Decisions Made, Things Built, Problems Encountered, Blockers, Next Steps) and you get summaries that are useful rather than just shorter versions of the raw logs.