Internal system names, providers, and exact numbers have been abstracted or generalized for confidentiality — the architecture patterns and trade-offs described are accurate.
Context
Property records are fetched through a national electronic registry network. A single request can come back with several PDFs attached: the record itself, a lien certificate, and — mixed in with them — a receipt for the notary fees.
Two problems. The customer was receiving a pile of separate files instead of one document. And the fee receipt is an internal accounting artifact; it has no business being in what the customer opens.
Merging is the easy half. The hard half is deciding which attachment is the receipt, because the answer isn't in the metadata: filenames are opaque, and the registry's type field doesn't distinguish them reliably. The distinction only exists in the content — and even there it's slippery. A fee receipt often carries the registry's stamp and cites the record number, so keyword matching on "certificate" or "record" classifies it as exactly the thing it isn't. What separates them is function: certifying content versus proving payment.
That made content classification an LLM job. It also made the failure mode the whole design problem.
The asymmetry that drives the design
The two ways this can be wrong are not comparable:
- Keeping a receipt that should have been dropped — the customer sees one extra page. Cosmetic.
- Dropping a certificate that should have been kept — the customer receives a document that is silently missing the thing they paid for. Nobody notices until someone tries to use it.
So the classifier is deliberately biased. It is not asked "which of these is the receipt"; it is asked to produce a type and a confidence, and an attachment is excluded only when it comes back as a receipt with high confidence. Every other outcome keeps the file:
| Classifier outcome | Action |
|---|---|
| receipt, high confidence | Exclude |
| receipt, medium/low confidence | Keep |
| certificate or indeterminate | Keep |
| Empty text (scanned PDF, no text layer) | Keep — LLM never called |
| Malformed JSON, timeout, provider error | Keep, log the exception |
There is a second guard behind that one. If every attachment classifies as a receipt, the merge ignores the classification entirely and merges all of them. That should never happen — a response always contains at least one certificate — so it means the classifier is wrong or the response is shaped in some way nobody anticipated. Either way, emitting an empty PDF is worse than emitting an extra page.
Text extraction is the cheap path on purpose: the first three pages via PyMuPDF, no OCR. A scanned PDF with no text layer yields an empty string, which routes to "indeterminate" and keeps the file. OCR coverage is left as a deliberate gap rather than a silent one.
Architecture
cron pass
│
▼
registry response ──► N attachment URLs
│
▼
for each: download ──► extract text (3 pages, no OCR)
│
▼
LLM classifier ──► {type, confidence}
│
▼
receipt + high confidence?
┌──────┴──────┐
yes no
│ │
drop keep
│
┌───────────────┴───────────────┐
▼ ▼
all dropped? merge all merge kept ──► single PDF
│
▼
ResultadoMesclagem
┌──────────────┬───────────────────┴──────┐
▼ ▼ ▼
SUCCESS TRANSIENT FAILURE PERMANENT FAILURE
│ │ │
mark answered stay queued, mark error +
+ notify retry next pass alert with link
(no alert)
The classifier runs synchronously, so it needs a leash
This path runs inside a scheduled pass, not a worker queue. The LLM client's default timeout is around ten minutes, and the stack falls back to a second provider when the first one fails — so an unbounded call could hold the container for a long time per attachment, on a job that has other records waiting behind it.
A 30-second per-call timeout bounds the worst case. Combined with fail-open classification, a slow or dead provider degrades into "merge everything" rather than into a stalled cron.
Failure taxonomy instead of a boolean
The merge helper originally returned true/false, which forced the caller to treat every failure identically. That's what made a one-minute network blip freeze a request permanently.
It now returns an enum, and the caller maps each case to a different outcome:
- Network or HTTP failure → transient. The record stays queued and the next scheduled pass retries on its own. No alert — there is nothing for a human to do while automatic retries are still running, and firing an alert every hour would train people to ignore the channel.
- File arrived but won't open → permanent. Re-downloading fetches the same broken bytes. Mark the record as errored and alert, because only a human can resolve it.
- Unknown error → permanent, on purpose. Retrying an unexplained failure every hour generates load with no prospect of resolving, and keeps the record out of the operators' view. Better to surface it.
- No attachments → its own state, not an error.
Making the alert actionable
The alert was sending only the record number, while every other message in the same channel carried a link. The recipient had to go and find the record by hand — which is precisely the work the alert existed to save. It now carries a deep link into the backoffice.
One bug worth keeping a note about
The registry returns presigned download URLs with the query-string & HTML-escaped as &. Used as-is, the signature doesn't validate and the download 403s. Unescaping the URL before the request fixes it — a five-character change that took much longer to find than to make, because the failure looks like an expired-credentials problem rather than a string-encoding one.
The download itself retries three times with backoff on 429 and 5xx, with separate connect and read timeouts.
Trade-offs
Fail-open classification, not fail-closed. The classifier is allowed to be wrong in the direction of doing nothing. Some receipts will reach customers. The alternative — a classifier that drops on uncertainty — would occasionally destroy the actual deliverable, and would do it silently. Wrong-but-visible beats wrong-but-invisible when the artifact is what someone paid for.
LLM for classification, deterministic code for everything else. The model decides one thing: what kind of document this is. It doesn't decide whether to merge, what the resulting status should be, or whether to alert. Those are ordinary branches with ordinary tests, so a bad model response can only ever produce an extra page — never a wrong workflow state.
Content classification, not metadata. Reading the file is slower and costs a model call per attachment. Filename and type metadata were tried first and are simply not reliable here; a receipt and a certificate are distinguished by what they say, not by how they're labelled.
Synchronous inside the cron, not moved to a queue. A worker queue would remove the timeout concern entirely and is the better long-term shape. Keeping it in the scheduled pass avoided introducing a new async surface, a new failure mode, and a new thing to monitor for a job that runs on a fixed cadence and finishes quickly. The 30-second leash is the cost of that choice, and it's explicit rather than assumed.
Three pages of text, no OCR. Covers text-layer PDFs, which is nearly all of them, at negligible cost. Scanned attachments fall through to "keep", which is the safe direction. Adding OCR would improve precision on a small tail and is a known follow-up — the gap is documented in the code rather than left for someone to discover.
Outcome
- One file. A multi-attachment response becomes a single PDF, with internal fee receipts excluded from what the customer opens.
- No silent loss. Every classifier failure mode — empty text, low confidence, bad JSON, timeout, provider outage, all-receipts — resolves to keeping content. The worst case is an extra page.
- Retries that actually retry. Network failures stay queued and recover on the next pass instead of freezing the record, which is what the old boolean return caused.
- Alerts worth reading. Only permanent failures alert, and the alert links straight to the record.
- 19 tests across the merge helper, the classifier predicate, and the status-and-alert mapping — including the all-receipts fallback, the transient-versus-permanent split, and the presigned-URL unescape.