Building Gary, the AI Dungeon Master Assistant

I’ve been running D&D campaigns since 2018. I love designing worlds and improvising at the table, but some parts of prep I enjoy less: wordsmithing a description that hints at a trap without revealing it, iterating on an idea until it clicks, writing in-world correspondence with hidden clues. Since ChatGPT, I’ve been using LLMs for those tasks, but I’ve found that giving them the right context is time-consuming.

I keep my notes in Obsidian, in plain markdown. Even with Claude’s Projects feature, I have to manually sync files to the web UI. Missing context leads to inaccuracies you don’t catch until you’re at the table.

So I built Gary: an AI writing assistant that lives on my laptop, reads from my Obsidian vault, helps me iterate on ideas, and writes polished text back into my files.

Gary isn’t meant to be a D&D themed slop generator. It’s not supposed to dump huge chunks of text you will never read on your disk. Gary doesn’t replace my creativity - it enables it.

My Goal

Past side projects taught me that without clear and realistic goals, I end up with nothing to show for my time. I decided to do better this time:

• I wanted to build a usable version of Gary in 5 days and use it before my next Saturday session
• I wanted to learn something new in the process of building it
• And I was ok with spending up to $100 on it

With that, I had a close enough approximation to not only what I wanted, but how I wanted to build it. Some more research and practical considerations later, I had a good idea of the stack I wanted to use:

• I decided to build a desktop app rather than a cli tool, mostly because I had never done that
• This gave me two options: Qt and Electron. Other less well-known frameworks also came up, but I decided that I would go with something I had heard of before
• Electron required me to use Typescript, which I only have cursory knowledge of. Initially that put me off, but I realized building a UI in a language not meant for it would end up as painful as learning a new language
• Besides, my decision to have a usable version in 5 days already meant I would heavily rely on AI to write most of the code
• But if I used AI to write code in an unfamiliar language, I would insist on clean, well-tested code. I would refactor regularly and challenge the AI when its decisions didn’t make sense

The Spec

My first instinct once I had made these decisions was to write a design doc. This is what I did at AWS when handing off a chunky project to somebody more junior, so why should AI be different?

However, as I began writing the doc, I realized I was spending too many tokens on explaining my decisions, rather than clarifying them. I deleted everything and started over, focusing on really trying to express in words the implicit assumptions I had made so far.

This led me to a clear list of what was in and out of scope for the first version of Gary:

1. Gary is an Electron app written in Typescript
2. Gary’s user mainly interacts with it through chat
3. It should start by loading an Obsidian vault
4. It should not load things into context until they are needed
5. It should use agentic search to find relevant files
6. It should not focus on rendering files for now

… and so on. I ended up with four main user flows, details about file formats, constraints, and an explicit list of what was out of scope.

Much has been written about spec-driven development, from heralding a codeless future to lamentations about return to Waterfall. I wouldn’t go far in either direction. Specs do what good writing always does: forces clarity and exposes which ideas were less solid than you thought.

The Plan

Within a couple of hours, I had clear goals and a product spec. I was ready to (watch AI) write code. Or was I?

I decided to spend some tokens on Claude’s plan mode. I committed the spec to source control and asked Claude to develop an implementation plan, breaking down the spec into milestones and iterations.

It was… Underwhelming. In the first version of Claude’s plan, Gary was divided into 3 milestones (Claude, taking a page out of every middle manager’s book, really likes 3 point plans). The first milestone included the landing page, vault loading, chat, AI integration (without tools) and a bunch of other bells and whistles.

I told it that the spec and plan was all a junior dev would have to go on. That the plan should be detailed and clarify everything that isn’t reasonably clear, and result in deliverables that I could manually test and provide feedback on. That it should ask me questions about things it is not sure about. That seemed to do it, with Claude this time generating 13 simple milestones like “Electron Hello World” (M1) and “Read-only tools” (M6).

I edited the plan a little bit further and committed it to the repo. I was ready.

Next post: what actually happened when AI started writing code, and how many times I have to ask for unit tests.

Usually people make websites because they have something to say that others might want to read. This is not true in this case, and to be perfectly honest I am putting words into this file because the default tutorial I am following had something about bananas and I found that boring.

The existence of this website was entirely motivated by my realization, a couple of years ago, that Icelandic websites use the .is top-level domain, and that i and s are the only two letters missing from my usual alias mentekid. I decided it would be nice to own mentekid.is, this way complicating my already hard-to-spell name even further.