Designing a Writing Collaborative Partner

Ask an LLM to write your childhood memory and you will get "bustling" markets and "humid" evenings: fluent, plausible, generic, and untrue. This is the failure mode of AI writing that matters most for personal and expert work. The value of a memoir is that your grandmother said that thing in that way; the value of a technical article is that an expert with specific scars is doing the arguing. Fluency is not the scarce resource anymore. Voice is.

This post describes Memoir, an agentic interviewer I built that helps you write a life story, a book, or an article in your own words. You talk to it, by live voice or text; it interviews you, remembers across sessions, and assembles prose composed only from what you actually said. It cuts and reorders; it never writes for you.

The problem with "AI writes it for you"

Many people want to write something long and never do. The blocker is usually not ideas; it is the writing process itself. Professional ghostwriters solve this by interviewing: they draw material out in conversation and shape it afterwards. Most people do not have access to one.

The obvious alternative, asking a model to write it, produces text that is not yours. And as generated text floods every channel, provenance becomes the interesting property: not "is this well written" but "did a person actually say this." I wanted to build the interviewer, not the ghostwriter, and to make the distinction enforceable rather than aspirational.

That framing produced three design goals:

• Keep the writer's voice. The AI never writes on the author's behalf. Not as a stylistic preference, as a hard constraint checked in code.
• Interview well. The craft of a good interviewer (go deeper rather than moving on, follow energy, ask about the specific afternoon and the specific object) is where the material comes from.
• Edit without rewriting. Point at gaps, contradictions, and thin sections. Propose; never overwrite. The author decides everything that enters the book.

The provenance gate

The central mechanism is a faithfulness check that runs in code, before any composed sentence reaches the author. We do not rely on prompting the model for correctness. Every meaningful word in a composed sentence must appear in the author's recorded speech. Small connecting words are allowed freely, since that is what makes rearranging possible; names and numbers must match exactly. If the model slips in a word the author never said, the sentence is blocked, the model is shown which word was invented and asked to retry, and a sentence that still fails is discarded rather than shown.

you said:   "So, um, we moved to Chicago when I was, I think, seven."
accepted:   "We moved to Chicago when I was seven."     (cut + reorder only)
rejected:   "We moved to bustling Chicago when I was             seven."                                    (adds "bustling")

In the interface, every sentence is a receipt: click it and you see the exact recorded words it was composed from.

A second, meaning-level check (an LLM judge looking for distortions that survive a word-level check, like a dropped negation or a flipped attribution) runs in evaluation and can be enabled at composition time. But the lexical gate is the guarantee, and it is deliberately boring code rather than a model behavior.

Where the autonomy is

The interview itself is human-paced. The autonomy runs between sessions. When a session ends, a multi-agent crew picks up the work unsupervised: an archivist files people, places, and events into a memory bank and composes new passages; a reviser notices when new material belongs inside a passage that already exists and proposes the merge as a diff; a critic re-reads the affected chapters against everything known and reports contradictions, timeline conflicts, and thin sections; a planner prepares the next session with a reason attached to every proposed topic.

The human-in-the-loop line is drawn precisely. Agents propose; the author disposes. Nothing enters the book until the author accepts it, the reviser's merges arrive as side-by-side diffs with "use this" and "keep what I had," and the reading view renders only accepted sentences.

An interesting property of this structure is that it works at two time scales with the same design. An article takes a session or two; a book takes months of sessions. Because memory, plans, findings, and spend all persist per project, the interviewer in session twenty can ask about the radio you mentioned in session two, and coming back after three months lands you on a worklist that says where you were and why.

Architecture

Memoir runs on Google Cloud: a FastAPI service on Cloud Run, Firestore for state, Gemini for every model call, and Google ADK running the live interviewer.

Two decisions did the most work.

The engine is vendor-free, and CI enforces it. The editorial system (provenance gate, composer, interviewer, crew, memory, planning, usage ledger) is written against a handful of Python protocols; nothing under engine/ imports a vendor SDK, and a portability test fails the build if that changes. Everything Google-specific lives in one adapter directory. This is what made the multimodal work cheap: the app supports typed text, a speech cascade (Gemini transcription and TTS around the text interviewer), and native duplex voice with Gemini Live, and adding the live mode was one adapter with no engine changes. Whichever mode you talk in, both sides of the conversation are persisted verbatim, so every mode leaves the identical provenance record the gate needs.

The agent craft is markdown, not code. The interviewing technique, the editing constitution, the critic's rubrics, and the planner's heuristics live in a skills/ directory as plain text, selected by what the author is making: a life story gets a storytelling interviewer that asks about the kitchen of the old house; an article gets one that asks what a skeptic would say. This turned out to be the most reviewable part of the system. Behavior changes are diffs to prose, and a non-programmer can read exactly how the interviewer is supposed to behave.

One lesson worth stating directly: enforcing a restriction requires owning the agent loop. The provenance gate has to sit between the model and the author, inspect every sentence before it is shown, and send failures back. A lower-level agent library like ADK, which hands you the runtime and lets you decide what happens between turns, makes that possible. Fully packaged autonomous harnesses that own their loop are the right tool for other jobs, but a product defined by what the agent may not do needs the loop in the product's hands.

Evaluating a system whose claim is faithfulness

A system whose central claim is "your words only" should be able to prove it, so evaluation is part of the product rather than an afterthought. The lexical gate re-runs as a regression eval, with a semantic judge on top. The critic is scored for precision and recall against contradictions planted in synthetic authors whose biographies I wrote, so recall has ground truth. And a judged end-to-end harness drives whole journeys through the real API with model-played authors and asks, for every interviewer turn, the question a user would ask: given what this person is making and what they just said, is this a reasonable thing to say next? That last layer exists because every serious defect in the project's history was a failure of my assumptions, invisible to the unit tests that encoded those same assumptions.

What's next

The research question under this product is elicitation quality: does a good interviewer actually surface material a person would not have written unprompted? That needs a study, not a script. The application I keep coming back to is also the obvious one: elders, oral history projects, and people with a serious illness who want their story recorded, in their own words, while they can still tell it.

The scarce thing was never fluent text. It was your voice, on the record, kept.