I wonder if this is a case similar to chess, where LLMs kinda suck, but other models might be viable.
AccountingBench: Evaluating LLMs on real long-horizon business tasks
31–40 of 154 posts
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#32Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…
It gets it right for me... https://chatgpt.com/share/687e8c28-7714-800c-abf4-e9cd3ce87b...
Ah, wouldn’t be an LLM discussion thread without one of these “it works/doesn’t” conversations.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#33Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…
> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong]
Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous question without context anyway.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#34Absolutely love the UI!
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#35So there exists a 'Excel World Championship': * https://en.wikipedia.org/wiki/Financial_Modeling_World_Cup * https://www.cbc.ca/radio/asithappens/2024-excel-world-champi... Can't wait for this to start having 'e-sports' tournaments. :)
Sadly I did have time to find the parody video from 2019:
https://www.youtube.com/watch?v=ICp2-EUKQAI
And the not-parody: https://www.theguardian.com/australia-news/2023/dec/15/you-d...
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#36But can't it, literally, hallucinate raw data at any point in the run?
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#37Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those.
Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came in. Interestingly, the failures aren’t exclusively a context length problem, as we reset the context monthly (with past decisions, accruals/deferrals, and comments available via tool calls) and the types of errors appear to be more reward hacking vs pure hallucinations.
Accounting is very interesting in an RL-first world as it is pretty easy to develop intermediate rewards for training models. We are pretty sure that we can juice the performance more with a far more rigid scaffold, but that’s less relevant from a capabilities research perspective. We’re pushing down this research direction and will see how it goes.
Let us know if you have any questions!
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#38A serious problem for many accounting start ups who so far faked it till it will work. In other words, they still need to do more manual labor than they thought. They will never be profitable and it will take years, if ever, until AI will substitute the local accountant.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#39Hmm will openAI dogfood their own accountability with software like this ? Curious to know if they’ll be able to take this bet on their own money related software
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#40Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it.
Bookkeeping for my small business runs into the tens of thousands of dollars every year, and the amount of human error associated with processing assorted ecommerce and other transactions is astounding, even after extensive planning and SOPs.
The other pain point is Quickbooks. The tool is so sprawling and complex that half the time support agents can't figure out what's wrong. The fact that Intuit jacks up the price every year for this POS is very irritating. They get away with it because they are practically a monopoly, with most small business CPAs locked into their ecosystem.
Hope your team can work out the performance issues. Alternatives to the current bookkeeping options are sorely needed.