Designing Memoir: An AI Interviewer That Never Writes for You

The quote "Writing is thinking. To write well is to think clearly. That's why it's so hard," by David McCullough captures how writing is both a critical skill and a vehicle for thought. Yet, as AI agents are now able to generate endless reams of text, there is growing angst around the prevalence of careless or mindless AI slop and how it is negatively impacting human information spaces. As a technologist who has been writing for a few years now, I frequently evaluate the "role" of AI/agents as I write and I tend to group its use into two main buckets. On one hand, I find them useful as a research and thought partner (find x, critique this paragraph) and as a helpful editor (scan this for grammar, punctuation and readability issues). Where I tend to draw the line is using them for wholesale generation (create a draft and publish it). The latter, while very tempting, is perhaps not a good idea for several reasons. First, the resulting quality is often poor - it includes LLM-isms (its not this its that, the throughline is x, ...) that have come to deeply bother readers. Second and more importantly, it robs me of the exercise in thinking and collating my thoughts, and robs the reader of insights and personal experience that could not possibly be generated by any AI model.

Given all of these, how then should we use AI in writing, if at all? I think a good writing tool is one that supports thinking, one that will not readily degrade quality through average writing/prose and one that excels in teasing out my best ideas.

This post describes the first version of something I built to help with this - Memoir. Memoir is an agentic interviewer that helps you write a life story, a book, or an article in your own words. You talk to it, by live voice or text; it interviews you, remembers across sessions, and assembles prose composed only from what you actually said. It cuts and reorders; it never writes for you.

The problem with "AI writes it for you"

They say you should always build for yourself, aka be your first customer. I have always wanted to write out stories across my life experience - growing up, living on 4 continents, fun and challenging childhood stories that I worry are slowly being forgotten. I want to write it, write it well - so it reminds me of where I've been and also is there so I can share it with my kid when he gets interested in the details. But writing at that scale is long and hard. A few years ago I toyed with the idea of getting an interviewer or ghostwriter to interview me and help make progress on it one day when I had more time.

In theory, AI can help with this. However, without the right structure, context and the right UX, some suboptimal outcomes can arise. A naive LLM setup will happily produce text that is not yours. And without provenance, the question stops being "is this well written" and becomes one you cannot answer: "did I actually say this?" To succeed - to enable writing - this requires the right design: the right agent paired with the right experience, ideally voice based, mimicking what a clever, attentive interviewer might pull off.

For these reasons, Memoir is based on three design goals:

• Keep the writer's voice. The AI never writes on the author's behalf. Not as a stylistic preference, as a hard constraint checked in code.
• Interview well. The craft of a good interviewer (go deeper rather than moving on, follow energy, ask about the specific afternoon and the specific object) is where the material comes from.
• Edit without rewriting. Point at gaps, contradictions, and thin sections. Propose; never overwrite. The author decides everything that enters the book.

How the whole thing works

Memoir is a web app organized around a loop of short sessions:

1. You talk. Start a project (a life story, a book, an article) and the interviewer asks questions on the session screen, by live voice or text. Everything both sides say is kept verbatim.
2. Memoir works. When you end a session, a set of agents files what you said into a memory bank, composes draft passages strictly from your recorded words, critiques what exists so far, and plans the next session.
3. You decide. In the manuscript view, every drafted sentence carries its evidence: click it and see exactly what you said. You accept or reject each passage, and a worklist proposes what to do next, with a reason attached.
4. Repeat. Sessions accumulate into chapters. A reading view shows only what you have accepted, and exports it as EPUB, Markdown, or print-to-PDF.

The rest of this post zooms into the two parts of that loop that carry the design: how "only your words" is enforced, and what the agents do between sessions.

Can "never writes for you" really be enforced?

You cannot prompt a model into never inventing; I tried, and instructions alone failed constantly. So Memoir checks it the way a compiler checks types: in code, before anything reaches you.

The check runs on every composed sentence before the author sees it. Every meaningful word in a composed sentence must appear in the author's recorded speech. Small connecting words are allowed freely, since that is what makes rearranging possible; names and numbers must match exactly. If the model slips in a word the author never said, the sentence is blocked, the model is shown which word was invented and asked to retry, and a sentence that still fails is discarded rather than shown.

you said:   "So, um, we moved to Chicago when I was, I think, seven."
accepted:   "We moved to Chicago when I was seven."     (cut + reorder only)
rejected:   "We moved to bustling Chicago when I was             seven."                                    (adds "bustling")

In the manuscript view, every sentence is a receipt: click it and you see the exact recorded words it was composed from.

A second, meaning-level check (an LLM judge looking for distortions that survive a word-level check, like a dropped negation or a flipped attribution) runs in evaluation and can be enabled at composition time. But the lexical gate is the guarantee, and it is plain deterministic code, not a model behavior.

What the agents do between sessions

The interview itself is human-paced. When a session ends, a multi-agent crew picks up the work unsupervised: an archivist files people, places, and events into a memory bank and composes new passages; a reviser notices when new material belongs inside a passage that already exists and proposes the merge as a diff; a critic re-reads the affected chapters against everything known and reports contradictions, timeline conflicts, and thin sections; a planner prepares the next session with a reason attached to every proposed topic.

Agents propose; the author disposes. Nothing enters the book until the author accepts it, the reviser's merges arrive as side-by-side diffs with "use this" and "keep what I had," and the reading view renders only accepted sentences.

The same design works at two time scales. An article takes a session or two; a book takes months of sessions. Because memory, plans, findings, and spend all persist per project, the interviewer in session twenty can ask about the radio you mentioned in session two, and coming back after three months lands you on a worklist that says where you were and why.

Architecture

Memoir runs on Google Cloud: a FastAPI service on Cloud Run, Firestore for state, Gemini for every model call, and Google ADK running the live interviewer.

Two decisions did the most work.

The engine is vendor-free, and CI enforces it. The editorial system (provenance gate, composer, interviewer, crew, memory, planning, usage ledger) is written against a handful of Python protocols; nothing under engine/ imports a vendor SDK, and a portability test fails the build if that changes. Everything Google-specific lives in one adapter directory. This is what made the multimodal work cheap: the app supports typed text, a speech cascade (Gemini transcription and TTS around the text interviewer), and native duplex voice with Gemini Live, and adding the live mode was one adapter with no engine changes. Whichever mode you talk in, both sides of the conversation are persisted verbatim, so every mode leaves the identical provenance record the gate needs.

The agent craft is markdown, not code. The interviewing technique, the editing constitution, the critic's rubrics, and the planner's heuristics live in a skills/ directory as plain text, selected by what the author is making: a life story gets a storytelling interviewer that asks about the kitchen of the old house; an article gets one that asks what a skeptic would say. This turned out to be the most reviewable part of the system. Behavior changes are diffs to prose, and a non-programmer can read exactly how the interviewer is supposed to behave.

One lesson worth stating directly: enforcing a restriction requires owning the agent loop. The provenance gate has to sit between the model and the author, inspect every sentence before it is shown, and send failures back. A lower-level agent library like ADK (or similar agent frameworks), which hands you the runtime and lets you decide what happens between turns, makes that possible. Fully packaged autonomous harnesses that own their loop are the right tool for other jobs, but a product defined by what the agent may not do needs to own the loop itself.

Evaluating a system whose claim is faithfulness

A system whose central claim is "your words only" should be able to prove it, so evaluation is part of the product rather than an afterthought. The lexical gate re-runs as a regression eval, with a semantic judge on top. The critic is scored for precision and recall against contradictions planted in synthetic authors whose biographies I wrote, so recall has ground truth. And a judged end-to-end harness drives whole journeys through the real API with model-played authors and asks, for every interviewer turn, the question a user would ask: given what this person is making and what they just said, is this a reasonable thing to say next? That last layer exists because every serious defect in the project's history was a failure of my assumptions, invisible to the unit tests that encoded those same assumptions.

Final THoughts

Building this has also made one thing clear: a tool like this does not save the writer time. You spend more hours talking than drafting would take. The bet is on quality - that a good interviewer pulls out material I would not have written on my own. I do not know yet whether that bet pays, and I have already seen failure modes to tune out, like stale follow-up questions the transcript had already answered.

So the plan is to dogfood it on my own upcoming writing, and to work through the open questions this build surfaced: how to grow robust evals from my own transcripts, the right artifact format for books so a draft exports cleanly to toolchains like Quarto, the right mix of models per agent (Gemini 3.5 Flash has been surprisingly capable for most parts; stronger models likely earn their cost in the critic), and better speech recognition and synthesis for a true voice cascade.

Memoir is a hackathon-sized first version with rough edges - the demo deployment is a single shared workspace, for one. You can try it live and read the code on GitHub. If you do, I would like to hear what breaks.