Introducing the Nagaris AI Accounting Index

We tested 22 AI model setups on 112 Australian accounting questions and marked all 1,251 answers against question-by-question marking schemes. Here's what we found, and why Ask Nagaris doesn't make you choose a model.

Nagaris · October 2026

Every few weeks a new AI model arrives with a new leaderboard to prove it's the best. We read them all. And for a long time we couldn't answer the question accountants kept asking us: which model and thinking level should I actually use?

The big leaderboards are excellent at what they measure: coding, maths, science, general reasoning. None of them tells you whether a model knows that NT Picnic Day moves a super deadline, or that equipment sold with a Tasmanian farm attracts duty. We couldn't find a published benchmark built for Australian accounting practice.

So we built one. It's called the Nagaris AI Accounting Index, and this is the first edition.

What we tested

The Index has 112 questions across the seven areas of work our firms ask Ask Nagaris about every day:

• tax, entities and super
• state revenue and property
• employment and workplace
• accounting and systems
• finance and business advisory
• governance and practice compliance
• knowing which specialist a question belongs to, and when to stop and refer

39 of those questions are deliberately hard: the kind a senior manager would want to check twice. The other 73 are everyday questions every model should get right.

We put them to 22 model setups: 10 models from the major AI developers, several at more than one thinking level. That produced 1,251 answers. We judged every one on three things a firm cares about: is it right, what does it cost, and how long does it take.

How we built the questions

Every scenario is invented. No client data went anywhere near this.

The hard part was the answer keys. A benchmark is only as good as its answers, so every expected answer was worked out again from the primary source: the legislation itself, ATO guidance and the state revenue offices. That meant reading the Fair Work Act for redundancy and notice, the super guarantee rules as amended for Payday Super, and the Duties and Land Tax Acts of several states.

Then we tried to break our own work:

• Three separate reviews of the first draft.
• Every question solved again from scratch, without looking at the answer key, and compared.
• A second, independently built version of the benchmark, compared line by line with ours.

That process found real errors in our own answer keys, including a Victorian land tax question and a Tasmanian duty question. We fixed them before a single model was tested. We'd rather publish that than pretend we got everything right first time.

How we marked the answers

Each question has its own marking scheme, written like a senior's review notes. The headline answer carries two to three times the weight of the supporting points, so a well-written answer with the wrong number can't score well.

We started with the automated judge in our testing platform, but it boiled each scheme down to a single pass or fail and left 35% of answers unmarked, so models were being compared on different questions. So we re-marked everything.

All 1,251 answers were marked against the full schemes by Claude Opus 5.5 High, an AI model: 2,789 separate marking decisions in total. Model names were hidden and answers shuffled, so the marker never knew which model wrote what. Because Claude models were among those tested, we checked for favouritism; if anything, the marking was stricter on them. On the 811 answers both markers did mark, they agreed on whether the headline answer was right 85% of the time.

What we found

One model was accurate, cheap and fast all at once. GPT-6.1 Sol at its lowest thinking level scored 98 out of 100 on the hard questions and got 87% of them completely right. It cost about 1.4 Australian cents per question and took about 14 seconds. The highest scores (99 to 100) came from setups costing 52 cents to $1.36 per question and taking two to three minutes per answer, and the top scorer finished only 28 of the 39 questions.

More thinking didn't mean better answers. Maximum thinking took between 3 and 34 times as long per answer as the same model's lowest setting. For the strongest models it added a point or two at most. GPT-6.1 Sol at maximum thinking cost seven times as much, took seven times as long and scored no better. And thinking harder didn't rescue the small models: Claude Haiku at maximum thinking took nearly three minutes an answer and still scored 68. Five of the seven maximum-thinking setups didn't finish every question.

Smaller, cheaper models look fine until the work gets hard. On everyday questions, seven of the eight setups we ran scored above 90. On the hard set, the smaller, faster models fell to between 58 and 76, and they got only a quarter to a half of hard questions completely right. Those are the questions where a wrong answer costs a client real money.

State taxes are where models fall over. Duty and land tax rules differ in every state and change often, and that's where the gap opened widest. The stronger models scored in the high 90s there. The small models scored between 32 and 44.

Even strong models have blind spots, and they're different ones. Claude Opus scored 100 on state revenue but 79 on employment questions, where GPT-6.1 Sol scored 94. That's why we don't rely on one model.

One question beat most models. A Victorian employer pays wages on Thursday 30 July 2026. Under Payday Super, the super has to reach the fund within seven business days. The catch: NT Picnic Day, Monday 3 August, is a public holiday for a whole territory, so under the ATO's rules it isn't a business day. The deadline is Tuesday 11 August. The ATO's own worked example for a 30 July 2026 payday gives the same date. Not one of the 14 setups that answered it got the date right. Most said 10 August, four counted calendar days and said 6 August, and one said the same day as payday. That's exactly why a person still reviews the work, and why we test. Ask Nagaris closes this gap with checked, up-to-date knowledge bases and lookup tables for rates, thresholds and business days, which makes it more accurate than any of these models on its own.

The full results

Hard set, 39 questions, ranked by score. "Fully right" is the share of questions that got full marks.

Where a setup didn't finish, its score covers only the questions it answered. Seconds are the average per question from one run. 

Why Ask Nagaris doesn't have a model picker

Choosing an AI model shouldn't be part of an accountant's day. Working out which model knows Australian tax is our job, not yours.

Behind every answer in Ask Nagaris is a routing setup built from this research. The model to answer your question is chosen on accuracy, cost and speed together. Behind it sits a chain of fallback models from different labs, each put through the same 112 questions, so a second answer is less likely to share the first one's blind spots. When a new model is released that we think could be an improvement, it goes through the Index. If it beats what we use on accuracy, speed and cost, it takes over. You just ask the question. Choosing, testing and re-testing the models happens behind the scenes, and it never stops.

What comes next

This is version one, and we're open about its limits. Each question was answered once, so a gap of a point or two between models is a tie, not a win. Several maximum-thinking runs didn't finish, and the everyday set was only run on eight of the cheaper setups. 

Australian accountants deserve tools tested on Australian work, and this is our start on that.