Openly tested
Open source benchmarks
How well does Bilanzo book? Fiduciary offices run every version of Bilanzo against the Bilanzo reference set: test documents with expected values derived from the text of the law (source given for each case). Issued by Bilanzo, not by an authority. The documents are invented but realistic, the postings are calculated by hand. Only this test set counts, so that all runs are comparable. This page shows what comes out of it, even when it does not look good.
Bilanzo baseline test: language models compared
Measurement by Bilanzo on 2 November 2026, not confirmed by fiduciary offices. 420 accounting cases in 13 classes, the same tasks and the same aids for every participant.
Level 1: pure model results
Each model solves the tasks on its own. It receives the tasks and the aids (for example exchange rates, tax rates and fact sheets), but no knowledge graph and no access to any accounts. We count how many cases are strictly correct.
Claude Opus 5.5
80.2%
337 of 420 cases strictly correct
Claude Fable 5.1
75.5%
317 of 420 cases strictly correct
- Bilanzo model 1.1.0: not assessable, answers empty or in a different format (cause found, new measurement to follow).
- Base model 7B: everyday document coding 0 of 60.
| Class | Cases | Claude Opus 5.5 | Claude Fable 5.1 |
|---|---|---|---|
| Everyday document coding | 60 | 58 | 58 |
| Multi-step cases | 30 | 21 | 18 |
| Documents that cannot be posted | 30 | 30 | 29 |
| Depreciation | 30 | 27 | 28 |
| Payroll and social insurance | 30 | 30 | 29 |
| Payroll: occupational pension and tax at source | 30 | 30 | 30 |
| Closing: accruals and deferrals | 30 | 27 | 28 |
| VAT: mixed rates | 30 | 25 | 23 |
| Foreign currency | 30 | 26 | 15 |
| VAT: place of supply and reverse charge | 30 | 8 | 6 |
| VAT: net tax rate and flat tax rate | 30 | 2 | 3 |
| Closing: valuation | 30 | 29 | 27 |
| Company law and withholding tax | 30 | 24 | 23 |
| Total | 420 | 337 80.2% |
317 75.5% |
How we measured
- 13 classes, 420 cases, run of 2 November 2026.
- Tasks and expected values were frozen before the first run and sealed with SHA-256. The expected values were checked in a blind review: the reviewers first solved the cases themselves without seeing the expected values.
- Scoring is strict, following the frozen rule of the test set, by a fixed program and not by a language model. Before scoring, each of the 13 classes passed a cross-check: the expected values as answer score full marks, a deliberately falsified answer shows exactly the errors built in.
Known ambiguities
The test set includes VAT form box 200 under the effective method but not under the net tax rate method. If a model adds box 200 or leaves it out, the case is strictly wrong. The strict figure is the one that counts. The second figure only shows the size of the ambiguity if box 200 is not counted.
| Class | Model | Strict (counts) | Without box 200 |
|---|---|---|---|
| VAT: net tax rate and flat tax rate | Claude Opus 5.5 | 2 of 30 | 21 of 30 |
| VAT: net tax rate and flat tax rate | Claude Fable 5.1 | 3 of 30 | 17 of 30 |
| VAT: place of supply and reverse charge | Claude Opus 5.5 | 8 of 30 | 17 of 30 |
| VAT: place of supply and reverse charge | Claude Fable 5.1 | 6 of 30 | 12 of 30 |
Level 2: Bilanzo with knowledge graph
Here the product helps: Bilanzo works with the knowledge graph of the company Mediaquotes Jones Schibli. These results are therefore not the same as level 1.
Not yet measured, follows after training.
Claude is a trademark of Anthropic. Anthropic and Bilanzo are not in any partnership.
Current status
No confirmed run yet.
Until a fiduciary office has transmitted a run on the Bilanzo reference set and agreed to it, we show no figures and no badge here. Unconfirmed runs appear in the history further down as soon as there are any.
What is tested
For each document the benchmark compares account, contra account, VAT code, amount and period with the booking of the Bilanzo reference set, calculated by hand. Documents where Bilanzo should reject or ask count as correct when Bilanzo rejects or asks. If Bilanzo books a document in automatic mode without an approved knowledge chain, the whole run fails.
The case classes of the Bilanzo reference set are VAT standard rate, reduced rate, accommodation, documents with two rates, net tax rate, foreign currency, credit note, partial payment, capital expenditure and cases that are rightly stopped. Each class has a threshold; a class is passed when its rate reaches the threshold.
Who tests
Fiduciary offices that run the Bilanzo reference set in their own installation. Test set, case classes, weights and thresholds are fixed and the same for everyone; every version carries a checksum. A run only counts as confirmed once the fiduciary office has checked the result and expressly agreed to transmit it. Every run is stored with a checksum in a history that can only be appended to.
How to take part
- You install Bilanzo; the Bilanzo reference set is included.
- After every update you start the run in the settings under Benchmark. It runs on your computer, and nothing is posted.
- You check the result in the report document by document and, with your consent, send it to us as a file by e-mail. Only then does it appear here.
Your clients’ documents never leave your installation. You can make a run on your own documents locally; it stays with you, without a badge and without publication.
Take part as a fiduciary office The public repository with tool and rules will be linked here once it is open.
Without warranty
The results on this page are provided without warranty. They show how a specific version of Bilanzo booked the Bilanzo reference set in the installation of individual fiduciary offices at the stated time. They are no assurance for your bookkeeping and do not replace a professional review. The reviewing fiduciary offices provide their runs and confirmations voluntarily and accept no liability for them. The comparison models were tested with the same tasks and aids, without access to any accounts. Anthropic and Bilanzo are not in any partnership. The models named were tested as publicly available services.
