Notes · 25 September 2026
One reasonable line of code, eight wrong figures.
A shop owner exports the quarter's orders from Shopify and gives the file to an AI assistant that can run Python. The request is what an owner would write: the key numbers of the quarter, "exact numbers, for an investor presentation". The assistant returns a tidy report with 113 numbers.
The export in this test is synthetic but in the real Shopify format: 21,687 rows and 9,659 orders, with the mess a real export has, such as overlapping downloads, cancelled, test and staff orders, VAT-inclusive prices. Because the data are generated, the right answers are known.
What the recount found
Of the 113 numbers, 35 have a definition that separate checking code computes. 27 of them matched to the cent. 8 did not. The other 78 were listed with the reason they could not be recounted: a different definition, a split the checking code does not compute, or a description of how the data were cleaned.
All eight differences come from one line of the assistant's code:
q['refund_prod'] = q['Refunded Amount'] * q['Subtotal'] / q['Total']It takes a proportional share of shipping out of every refund. For the 364 full refunds that is right: they returned the shipping too. The 439 partial refunds in this export are goods only, and from them the line took shipping that was never refunded. Refunds for the quarter came out EUR 505.35 too low. Net sales came out too high in every month (by EUR 207.78, 149.19 and 148.38), by EUR 505.35 for the quarter, and by the same amount for the online-store channel.
Why re-reading would not catch it
- The line reads as a sensible assumption. The export cannot say whether a partial refund included shipping, so it had to be decided one way or the other, and here it was decided silently, in a code comment the report never mentions.
- A second look at the same code shares the same assumption. A recount by separate code that never saw the report does not.
- EUR 505 is small next to EUR 1.47 million. But a figure an investor is shown is either the number the data give, or it is not.
What I take from it
Before counting, write the definitions down and have the owner answer the ones the data cannot: is shipping part of a partial refund, is VAT in net sales, which date counts. Then recount every figure that can be recounted, and say plainly which ones cannot and why.