Bookkeeping looks like a perfect job for a language model. There is a lot of messy text: bank descriptions, receipts and invoices in several formats. Somebody has to read it all and decide what each payment was.
But bookkeeping has a property that most AI demos do not have. The answer must be exact. Every voucher, the record that moves money between accounts, must balance to the last øre. Years later, somebody must be able to see why it was booked that way. A result that is nearly right is simply wrong.
So we do not ask how to make the model perfect. We ask how to build a system that is reliable when one part of it is not. This post goes deeper into one of the four layers we work on.
Put the model in the middle
We place the model between two stages of ordinary, tested code.
Everything that can be computed gets computed before the model starts. Code pairs up transfers between a company’s own accounts. Code finds the documents that may belong to each payment, and code works out the opening balances. The model receives these as facts, so it does not spend its attention on arithmetic or on searching.
This also keeps the model’s input small and plain. The code that gathers data is separate, and row-level security limits what it can read. The workflow that hosts the agent, which is the model plus the tools it can call, only receives data. It has no database client and no credentials, so a confused agent has nothing dangerous to reach for.
Let the validator talk back
After the agent comes the second stage of code. The agent does not write to the ledger. It proposes a complete voucher, and a validator checks it.
The validator is a pure function with no I/O. The interesting part is what it returns. It does not only say no. It returns a structured error that the agent can read and act on:
// Simplified, with invented names. The real validator checks much more.
type VoucherLine = { account: number; amountInOre: number };
function checkBalance(lines: VoucherLine[]) {
const total = lines.reduce((sum, line) => sum + line.amountInOre, 0);
if (total === 0) return { ok: true };
return {
ok: false,
error: {
code: "UNBALANCED",
message: `The lines are off by ${total} øre. They must sum to zero.`,
},
};
}The agent gets the error back as a tool result, corrects the voucher and tries again. Rules that must always hold live in code, where we can test them. The agent does the part that needs judgment, and it can never talk its way past the rules.
Make “I don’t know” a good answer
A model that must always answer will guess, and a confident guess in a ledger is expensive. So we designed the system to ask instead.
When a payment is unclear, the proposal can be a question for the customer. The system treats that as a good result, not as a failure.
Completeness is enforced in the same spirit. The agent cannot finish a run while a transaction is unresolved. If the run ends with something uncovered, that transaction goes to a person. If the model call itself fails, the whole run fails, and we do not get half a batch that looks complete. The worst outcome we can design out is a payment that nobody looks at.
Treat the data as untrusted
A bank description is free text that somebody outside the company wrote. It can contain line breaks, Markdown or a sentence that reads like an instruction. To a language model, data and instructions are all just text.
So every free-text field goes into the prompt as an inert, quoted string, through one small helper. A payment message is part of our attack surface.
Choose the simple shape
Our first version had a coordinator agent that delegated work to a proposing agent. It was an appealing design, and the frameworks make it easy to build.
We removed it. Delegation exists to split context across large jobs, and our batches are small. One agent with a direct tool loop gave us no budget rules to tune and no two agents to keep in sync. If batches grow, we can add delegation back. Until then, the simpler system is easier to test and to reason about.
We apply extra model time in a narrow way too. A separate review pass looks only at the proposals that deterministic code marks as risky, so most of the work stays cheap and fast.
The accountant has the last word
A proposal from the agent becomes a voucher only when an accountant approves it. They see the proposed lines and the reasoning, and they can correct the proposal or send it back.
This is more than a safety net. It decides the whole shape of the product. The system’s job is to make an expert faster, and every proposal has to be easy for that expert to check. So a good question for any design here is this: how quickly can an accountant see that this is right?
If you want to work on problems like these, see our open roles.