Cases · Report check
AI sales report: 35 figures recounted, 8 wrong
Self-initiated case study using synthetic data in the real Shopify export format, so the right answers are known.
Goal
A shop owner asked an AI assistant for the quarter's key numbers, "exact, for an investor presentation", and needed to know whether the report could be trusted.
Challenge
The report looked tidy: 113 numbers, clean code, no visible errors. The assumption behind the error was written only in a code comment, never in the report.
Solution
All 113 figures were reviewed. The 35 with a definition the checking code computes were recounted from the same Shopify export by separate code that never saw the report, under definitions written down first: what counts as a sale, VAT, shipping, which date.
Outcome
27 of the 35 matched to the cent and 8 differed, all from one line of code that took a shipping share out of partial refunds. Net sales came out EUR 505.35 too high. The other 78 figures are listed with the reason they could not be recounted.
Note on this case: One reasonable line of code, eight wrong figures
From the case
The case, 2 pages (PDF)
Service
Report checks for your own files
The same method on your exports: definitions in writing first, then an independent recount.