Why I Didn't Build Rule Referee with AI

By Autumn Wohlwend, published 2026-09-20. An experiment with two memory structures, the research on RAG hallucination rates, and why a rules app needs verified answers instead of generated ones.

I was reading Why We Remember by Charan Ranganath PhD when I came across a passage about how the human brain handles exceptions.

The example was a penguin. Neural networks learn that birds have feathers, wings, beaks, and fly. Then you teach it a penguin is a bird; but a penguin swims. The network doesn't quietly add an exception. It forgets what it previously knew about birds to accommodate the new information. Ranganath calls this catastrophic forgetting, and it's one of the reasons AI systems trained on massive datasets still glitch on edge cases.

That wasn't the part that stuck with me though.

What stuck was the section on episodic memory. The hippocampus doesn't store every detail of your life. It works more like an index: it activates the right memory first, then retrieves the details from inside that specific episode. Flat recall searches everything at once; hippocampal recall narrows before it retrieves.

I kept thinking about what that distinction would mean for an AI system trying to answer questions accurately.

So I Ran an Experiment

I built two systems using OpenAI's Agent Builder and vector stores. Same model. Same data. Just different memory structure.

Flat RAG: the system searches all documents at once, pulls the most relevant chunks, and generates a response.

Hippocampal RAG: the system selects the most relevant episode first, then retrieves only inside that episode before responding.

I asked both systems the same questions and measured episode-clean recall accuracy; meaning the answer had to stay true to a single source, not blend details from multiple ones.

  • 81% episode-clean recall accuracy with flat RAG
  • 94% episode-clean recall accuracy with hippocampal RAG

The number that mattered most wasn't the accuracy rate though. It was what happened when a detail wasn't in the memory at all. The hippocampal version said so. It didn't fill the gap. The flat version often did.

That distinction is everything when accuracy is not optional.

What the Research Actually Says About RAG

RAG, retrieval-augmented generation, is the architecture most AI rules tools use. The system retrieves relevant documents, then a language model generates a response based on what it retrieved. It reduces hallucinations compared to a model answering from training data alone. But it does not eliminate them.

The research on this is consistent and not particularly kind to RAG's reputation in accuracy-critical domains.

A 2025 Stanford study published in the Journal of Empirical Legal Studies tested legal AI tools specifically marketed as hallucination-free, including purpose-built RAG products. Those tools still produced incorrect or misgrounded answers on 17% to 33% of queries. One exceeded 34%.

17-33% of queries returned incorrect or misgrounded answers from legal AI tools marketed as hallucination-free. (Stanford, Journal of Empirical Legal Studies, 2025)

A study published on arXiv compared a RAG system to a chatbot that retrieved and presented actual source passages directly to the user without generating a response. The RAG approach produced a 27.5% hallucination rate. The passage-retrieval approach produced zero.

  • 27.5% hallucination rate when the system generated an answer
  • 0% hallucination rate when it showed the source passage instead

That second study is the one that clarified everything for me. The zero-hallucination result came from a system that did not generate responses at all; it just found the right passage and showed it. No synthesis. No interpretation. No gap-filling.

A 2025 paper on deterministic retrieval framed it this way: in deterministic domains, a 99% accuracy rate yields 0% operational trust. If users can't predict which answers are wrong, they can't trust any of them.

In deterministic domains, a 99% accuracy rate yields 0% operational trust.

Board game rules are a deterministic domain. The rule either says you can collect rent in jail or it doesn't. There is no probabilistic version of that answer.

What Rule Referee Does Instead

Rule Referee uses a deterministic search engine, not a generative AI model.

Every game in the app starts with the official rulebook and any publisher rulings. Those are parsed and structured into individual, human-verified answer sets; one per game, never pooled together. My experiment with hippocampal RAG pointed directly at this: keeping each game's knowledge base separate prevents the memory blending that causes flat retrieval systems to stitch details from different sources into a single answer.

When you ask a question, the engine matches it to a verified answer and returns it exactly as written. There is no generation step. Nothing is synthesized. The answer was written, reviewed, and tested before the game ever shipped.

Every game goes through a 260-question test suite before release; 200 core rules questions and 60 hobby-depth questions covering edge cases, named cards, and timing rules. Then a human spot-check before it goes live.

If Rule Referee doesn't have a verified answer for your question, it says so. It doesn't fill the gap.

That's the part I kept coming back to from my experiment. The hippocampal version earned trust not just by being more accurate, but by knowing what it didn't know.

Why This Matters for a Rules App Specifically

A wrong rule at the wrong moment doesn't just cause a bad answer. It ends the game. It causes an argument. It makes someone feel cheated. In the worst case, it's the last impression a game ever makes on a new player.

The board game community is also unusually skeptical of AI tools, and reasonably so. A lot of apps in this space use generative AI and don't disclose it. Players have learned not to trust answers they can't verify.

<a href="/lab/rule-referee">Rule Referee</a>'s answer to that skepticism isn't marketing. It's architecture.

No AI. No generation. No guessing. 65 games, fully offline, with answers sourced and verified before they ever reach your table.

Download Rule Referee on the App Store | Get it on Google Play