Yvette Teo

Building a receipt-to-expense workflow that knows when to stop

I wanted to remove the repetitive parts of expense logging without letting automation make financial decisions it could not properly explain.

The problem

Expense logging is simple until it isn’t. A clean receipt can be read, classified and recorded quickly. The difficult cases are the ones with an unclear merchant name, a foreign currency, a duplicate attachment or a total that does not match what the document appears to say.

I did not need a system that claimed to “fully automate finance.” I needed one that handled routine documents consistently and stopped when the evidence was weak.

The useful question was not “Can an agent process receipts?” It was “Does it know when not to proceed?”

Why persistent memory matters

I built this workflow in Hermes Agent, the AI agent I use through Telegram. What made it genuinely useful was not just receipt extraction. Hermes has persistent memory, so it can retain the operating rules I have taught it: how familiar expenses are handled, what might be a duplicate, and which uncertainties should come back to me. I do not have to explain the same context every time, but the workflow still stops when the evidence is unclear.

What I built

The workflow takes a receipt through a small sequence of checks before preparing an expense record.

1. Intake→Receive a document and preserve the original file.
2. Extract→Read merchant, date, amount and currency.
3. Check→Look for duplicate documents and missing information.
4. Decide→Pause for clarification if a material field is uncertain.
5. Prepare→Create the tracker entry with a link back to the source.

The important design choice is that “prepared” does not mean “approved.” The workflow reduces data entry, but financial judgment remains visible and reversible.

Where it failed

Unclear documents

OCR can produce a plausible-looking answer even when the source is hard to read. Rather than silently choosing the most likely value, the workflow asks for confirmation when the merchant, amount, date or currency is unclear.

Duplicates

A duplicate should not create a second expense, but it should not be deleted automatically either. The system reports the match and leaves the source document untouched.

Foreign currency

An amount can be extracted correctly while the accounting treatment is still unresolved. Currency and exchange-rate questions therefore remain explicit review points.

What I learned

01

Confidence is not evidence.
A model sounding certain is not a reason to bypass a financial control.
02

Exceptions define the workflow.
The routine path is easy; the value comes from treating ambiguous cases predictably.
03

Source links matter.
Every prepared entry should lead back to the document a human can inspect.

What I’m changing next

I’m testing a better review queue for ambiguous documents and a cleaner way to surface why the workflow stopped, without making the operator inspect technical logs.