Cases · Claude skills
Claude month-end reports for Shopify that show their work
Self-initiated case study on three synthetic Shopify stores with the traps of real exports. The same person wrote the skills, the tests and the graders, so it is not a blind test; every model answer and grade is public, so anyone can re-grade them.
Goal
A store owner asks Claude the same questions every month: how the month went, why less reached the bank than was sold, which products lose money. The answers have to mean the same thing next month, and the owner has to see what was counted.
Challenge
Opus 5.5 got the figures right in 53 of 54 runs, with the skills or without. What changed without them was the basis: a loss per unit on one store, the month's loss on another; payouts started from every order placed, test and cancelled ones included. On a 4,470-order store the 85 orders left out of the month's sales went unnamed.
Solution
Three skills (a monthly summary, a payout reconciliation to the bank, product margins) ask the owner's definitions once and print them under every answer. Every figure comes from a script. A number check traces every figure in the answer to the script's results, and a coverage check makes sure everything left out, disputed or not paid out is named. The export is checked first, so a renamed column or a new payment method is stopped or named.
Outcome
With the skills, 27 of 27 answers were on the stated basis (13 of 27 without) and named everything the owner should see (21 of 27 without). Of 24 mistakes planted in correct answers, the checks and a reviewer model stopped all 24. Of 8 changes to an export, the earlier version let 6 through silently; now all 8 are stopped or named. Not yet run on a real store's raw export: that is the first step of every order.
Evidence
Every answer and grade (GitHub)
The stores' data and truth files are public; the skills and the bench are private.
From the case
The case, 2 pages (PDF)
Service
Claude skills for your own files
Built on your export: your definitions asked once, every figure from a script, every answer and grade handed over.