EricTetzlaff

For fourteen years I managed community associations. Thirty-four of them at the end, with two thousand homeowners and fifteen million dollars in reserves. When a board asked whether they could fine a homeowner for a fence, the answer lived in a declaration written in 1987, amended three times since, sitting in a binder nobody had read end to end.

I was the retrieval system. I was also the ranking function, and the part that decided which amendment won.

That is not a metaphor I reached for later. When I designed BoardPath's answer scoring, the question I actually asked was: if this output were challenged in a legal setting, what would need to be documented about how it was produced? I had appeared in court on behalf of client associations. I knew what an evidentiary standard looked like, so I built to one.

The cost of being wrong was never abstract to me. A bad answer sent a board into a decision they could not take back, and it was my name on the advice.

Every architecture has a rule it depends on and cannot enforce. Mine used to live in README files: never import the AI layer into the offline core. That rule survives exactly as long as nobody is in a hurry.

So in ledger, my decision-record tool, one test does nothing but import the core and assert that the AI library never loaded. Cross the boundary and the build goes red before the code reaches anyone else's machine. The directory structure stopped being a convention and became something the test enforces.

The same move shows up everywhere I work, because it is the only version of a rule I trust. The memory tool opens the database it reads from in read-only mode, so a bug in it physically cannot corrupt the thing it is compressing. The private-notes repository ignores everything by default and allows files in one at a time, so a new file is private unless I say otherwise. I would rather make the wrong thing impossible than remember not to do it.

The useful question about an AI system is not what the model can do. It is which parts of the system are allowed to vary.

In Litt, my operations agent for small law firms, the router that decides which specialist handles an incoming signal is a plain Python dictionary. No model call. It returns the same answer for the same input, forever. The language model writes the narrative that a human reads, and it never decides what happens.

The consequence is the point: an outage costs prose, never correctness. A system where the model drafts and code decides degrades into something quieter and more manual. A system where the model decides degrades into something confidently wrong.

The failure I design against is not an error. Errors announce themselves. The failure is a fluent, well-cited, confidently wrong answer, produced because the system found two documents that disagreed and quietly picked one.

So BoardPath has a path that returns nothing. When the retrieved material will not support an answer, the request never reaches the language model at all, because a model handed thin evidence will write something anyway. Document ingestion has a gate that fails a corpus with a hole in it: six of seven expected articles is 85.7% of the document, and it is rejected rather than indexed. The confidence scorecard names the dimensions it could not measure instead of averaging them into a friendlier number.

Every one of those makes the product look worse in a demo. They are the ones I would defend hardest.

Most AI quality reporting is a language model scoring a language model, with nobody checking whether the scorer is any good. That produces a number, and a number is persuasive.

Before BoardPath's evaluation numbers counted for anything, I validated the judge against public human-labelled sets and recorded its own precision and recall. An evaluation I have not validated is a more expensive guess. When a measurement and a feature both looked broken, I fixed the measurement first, because until the instrument is trustworthy every decision after it is a coin flip with extra steps.

I turned off BoardPath's deterministic safeguards and ran the question path with nothing but the system prompt holding it together. It fabricated provisions that did not exist, fluently. That experiment is why the guardrails sit in code and not in instructions, and it is why I can tell you what the prompt is worth on its own: not much.

promptsmith, my prompt-engineering plugin, ships a set of deliberately broken test cases that the suite is required to fail. I built them after nine consecutive all-pass runs, because a suite that always passes and a suite that is broken produce exactly the same output. Now if the known-bad cases pass, the judge is wrong, and I find out from the harness instead of from a user.

I built a cross-encoder reranker for BoardPath. It measured well on clean synthetic documents. On real scanned governing documents it made retrieval worse, so I switched it off and left it in the codebase, dormant, with the evidence written next to it. The work was not wasted. The claim was.

Two of the architecture notes on this site carry dated corrections at the top, because the systems they describe changed underneath them and deleting the posts would have been tidier and less honest. The confidence package I published declares which of its dimensions are not being scored rather than quietly scoring them badly.

A number I withdraw is worth more than a number I defend past its evidence. Anyone can check the ones that are left.