The Method Stack

01The corpus, in numbers

Verified counts from the published shards and the extraction ledger.

The core batch is complete. The core DWD timesheet batch is ~40,000 unique documents (38,389 processed; the active queue is fully drained). The larger 72,719 figure counts pages, not documents — multi-page sheets and render variants inflate the page total roughly 1.8×. Accuracy has plateaued at this level: a 4th model and consensus voting both measured lower, and five LoRA fine-tunes all lost to the plain union.

Accuracy is reported three ways, honestly: 98.2% of readable fields recovered, 73.2% of sheets with all five fields correct, 37.2% strict character-exact. We score only cells where a true value exists — a model is never rewarded or punished for a genuinely blank cell.

02The pipeline, end to end

Each sheet — a phone photo or a scanned PDF — passes through the same stack.

1. Render & normalise. PDF pages rasterised at 150 dpi and scaled to ~2,000 px; photographs auto-rotated and deskewed only where a defect is detected — applied blindly, normalisation cost accuracy (see §6).
2. Three-model vision union. The page is read independently by three local vision models; their answers are reconciled field-by-field (see §3). No single model is trusted alone.
3. Second-pass geometry. The vision models return values but no coordinates. A separate Tesseract + OpenCV pass finds the real table grid and pins each extracted row to its actual pixels (see §4).
4. Compute & reconcile hours. Worked hours are recomputed from time-in / time-out / break with shift-aware logic (~139,500 rows computed; ~60,000 blank totals filled; ~20,000 discrepancies flagged, never silently zeroed) and cross-checked against the sheet's own stated total.
5. Classify & gate. A classifier confirms the page is actually a timesheet before its hours count — printed email threads and contracts are held out (see §8, learning of 4 Sep).
6. Score, route & publish. ~50% of sheets clear automatically at high confidence; ~50% with model disagreement or a weekday mismatch (1,534 auto-flagged) route to an HR reviewer, whose corrections feed the next evaluation.

03Why three models, not one

Any single vision model hallucinates on a bad photo — a smudged "3" becomes an "8", a blank cell gets a confident guess. Three models rarely make the same mistake. We read every page with all three and reconcile by first-non-null agreement keyed on the canonical date: a field is filled from the first model that read it, and a model's correct blank read is respected rather than overwritten by another's guess.

Diversity beats size. A 4th model (gemma3) dropped all-5 accuracy to 72.3%, and consensus voting to 70.5%. What earns an ensemble slot is a genuinely different failure profile — shared-bias errors are exactly what voting cannot fix.

04Where the boxes come from

The overlays are not decoration and they are not invented. Vision models return text with no coordinates, so a second pass — Tesseract word-boxes (TSV) plus OpenCV horizontal/vertical line detection — reconstructs the actual table grid, and each reconciled row is assigned to the grid band it physically occupies. A row we could extract but not locate is drawn as an inferred band and marked as such. Green = row extracted and date resolved; amber = extracted but the date could not be resolved from the sheet.

05What actually moves accuracy

Every gain below is measured on a held-out split built after the change was designed — so these are generalisation, not tuning-set inflation.

06What does not work

Recorded so no one retries them blind. Each was tested and refuted on held-out data.

• LoRA fine-tuning for full extraction — closed after five losses. Five runs, four base models, corpora from 33 to 2,606 files; every run lost decisively to zero-shot (best 12.2% exact vs 19.8–30.9% zero-shot on the same harness). The data-starvation hypothesis was falsified at 14× data. The frontier moves through the multi-model union, not a fine-tune.
• A 4th model, and consensus voting — both hurt. gemma3 dropped all-5 to 72.3%; consensus voting to 70.5%. Accuracy has plateaued at ~73% via the 3-model union.
• Global preprocessing. Re-encoding already-clean images cost −4.9 pt. Only touch what is measurably broken.
• Tesseract OSD for 0°-vs-180°. Wrong in both directions; its confidence is uninformative.
• Post-checks on legitimately-occurring patterns (e.g. equal in/out times): 0/24 precision — false precision is worse than none.
• Correcting the filename-period prior by its learned offset: −13 pt — the weekday constraint already absorbs that error.

07How the system keeps improving itself

Seven mechanisms are installed so the pipeline accretes gains and keeps its own claims verifiable.

1. Error-taxonomy-first loop. Every accuracy push begins by classifying errors (hallucinated / misread / missed) on held-out data — not by hypothesis. The two biggest gains both came from taxonomies; five plausible hypotheses tested without one were all refuted.
2. Frozen-split discipline. Every change is validated on a split built after it was designed. A +11.4 pt tuning-set gain was +3.7 pt clean — ~3× inflation, caught by the discipline.
3. Contamination gate. A content-hash audit across all splits runs before any cross-set number is quoted. Currently: zero duplicates spanning any two sets.
4. Per-corpus config, not tuned constants. K-of-N, per-field K and normalisation are keyed on source type (photo vs scan). Three corpora gave three different optima — one global pipeline is guaranteed wrong somewhere.
5. Ensemble prompt-parity assertion. Every union member must be on the same prompt version, asserted at assembly — a silent drift once cost 7.6 pt and was invisible in aggregates.
6. Build-time transforms re-run by default. Canonicalisation re-applies over new data automatically — a step that silently stopped once cost 20% of the corpus.
7. Artefact-verified agent work. Completion claims are checked against on-disk line counts, not exit codes — caught one false success and one silent stall in ~150 dispatches.

Where the frontier goes next. A photo-vs-scan router (both halves already exist); hard-example mining so ground truth grows where the model is weakest; a 4th diverse model only if it shows a different failure profile on the frozen set; and two narrow retargeted LoRAs (a row-detector head; scans-only) — each one run against the zero-shot bar, not a commitment.

08What we learned — the running log

Every item here changed the pipeline. Most were paid for in wrong answers first.

• 8 Sep · the core batch, counted honestly
  72,719 is pages; ~40,000 is documents.
  The headline was double-counting: multi-page sheets and render/mirror variants inflated the document total ~1.8×. A content-hash census confirmed 38,389 unique documents processed with the active queue fully drained — the core batch is complete, exactly as asserted.
• 8 Sep · duplicate attachments
  Gmail regenerates its attachment IDs on every fetch.
  Keying stored files on the volatile attachmentId re-ingested the same sheet as "new" each sweep. A stable content key (msg-id · filename · size) plus a one-time collapse removed 843 duplicate records (1,818 → 975 on the live portal), all owners preserved.
• 8 Sep · sweep, synchronised
  Sweep everyone from 1 Sept; skip the slow August backfill where it isn't needed.
  Not-yet-swept mailboxes are now swept from 1 Sept only (far faster) and tagged August-excluded; already-swept members keep their August. All 116 members swept concurrently so the whole workspace stays in sync.
• 27 Aug · the LoRA verdict
  The union beats a fine-tune.
  A task-specific LoRA lost to the plain three-model union on five separate evaluations. We retired the fine-tune: reconciliation across diverse models generalises better than one specialised head, and needs no retraining as templates change.
• 29 Aug · duplicate rows
  One date, written three ways, was counted three times.
  01/08/26, 01/08/2026 and 1/8/26 were splitting one shift into three rows and inflating totals. A canonical norm_date key now collapses format variants to a single row — the row still displays the sheet's own date text.
• 31 Aug · geometry, honestly
  Stop fabricating evenly-spaced boxes.
  Early overlays synthesised bands from the row count when no coordinates existed — they drifted off the real rows. Replaced entirely with the Tesseract + OpenCV grid pass (§4); a row that can't be located is labelled inferred, not faked.
• 1 Sep · the 24B bottleneck
  The heavyweight model was the whole cost.
  The 24B arbiter runs ~100 s/page and dominated throughput. When GPUs are contended we drop to a two-model 7B union (~3× faster) and restore the full three-model union the moment the fleet frees up — accuracy at will, speed when needed.
• 2 Sep · one GPU, many threads
  Parallelism can go backwards.
  Running the union's models concurrently on a single GPU thrashed VRAM and dropped throughput to near zero. The rule now: sequential models within a page when they share a GPU; parallel across pages and boxes when the models sit on separate GPUs.
• 3 Sep · the fleet is not uniform
  The Jetson is ~5× slower at vision than the Macs.
  Vision OCR is GPU-bound and the aarch64 Jetson is far slower per page — so batch extraction is balanced across the fast Mac boxes by residue-sharded work lists, with a supervisor that respawns any shard until its remaining count hits zero.
• 3 Sep · the network drops under load
  Every model call and Drive fetch now retries.
  Under heavy fleet contention, connections drop mid-request and silently lost a model's vote (or a whole file). Every OCR call and Google API call now retries with backoff, so a transient drop no longer costs an answer.
• 4 Sep · not everything is a timesheet
  A printed email thread will confabulate a perfect grid.
  A 125-hour "timesheet" turned out to be a printed Gmail approval thread — the model invented a uniform grid from prose. A classifier now gates every page: emails and contracts are held out of the hour totals and shown separately, so fabricated hours never reach a report.

09Principles we won't give back

• On-fleet only. Every page is read by local Ollama models on the Elitez fleet — no timesheet leaves the estate, and there is no per-page cloud cost.
• Reconcile, never overwrite. Disagreements between models, and between a computed total and a stated total, are surfaced for review — the pipeline flags, it does not paper over.
• Show the working. Every extracted row is drawn on the sheet it came from, at real coordinates, so any number can be traced back to the pixels that produced it.
• Honest denominators. Accuracy is measured only where a true value exists; date-resolution and confidence are reported per row, not as a single flattering headline.
• Nothing above the sheet is assumed. Hours notation, break handling and pay basis differ by row, by page and by person — the only safe inference is the forward test: does this sheet's own printed total reproduce under the assumed convention?