Cases · Report check

AI sales report: 35 figures recounted, 8 wrong

Self-initiated case study using synthetic data in the real Shopify export format, so the right answers are known.

8 of 35recounted figures were wrong, all from one line of code

Goal

A shop owner asked an AI assistant for the quarter's key numbers, "exact, for an investor presentation", and needed to know whether the report could be trusted.

Challenge

The report looked tidy: 113 numbers, clean code, no visible errors. It named its shipping assumption for full refunds and applied it, without a word, to partial refunds too. And the export cannot say whether a partial refund included shipping.

Solution

All 113 figures were reviewed against 38 definitions answered in writing before counting: what counts as a sale, VAT, shipping, refunds, which date. For this case an AI model answered them under the owner's delegation, and the document records that beside every rule; with a real client, the client answers. The 35 figures with a definition the checking code computes were recounted from the same Shopify export by separate code that never saw the report.

Outcome

27 of the 35 matched to the cent and 8 were wrong, all from one line of code that took a shipping share out of partial refunds. Net sales came out EUR 505.35 too high, and the same line moves 9 more figures the check could not cover. The other 78 figures could not be recounted, each listed with its reason: not among the agreed definitions, per month where the definitions give the quarter, defined differently, or describing the data clean-up. The owner gets the report with a note beside every figure (7 pages) and the verification document (15 pages).

Note on this case: One reasonable line of code, eight wrong figures