61–70 of 154 posts
Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
Earlier quoted context omitted.
> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous…
How can "which is the larger number" be an ambiguous question?
Earlier quoted context omitted.
I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.
Counterpoint: if you detect a human accountant doing this, you can take action against the human. Computers will never meaningfully take the blame, and unfortunately usually mean not blaming any human either.
Earlier quoted context omitted.
Human accountants are notoriously non-deterministic too, and any sufficiently complex accounting process contains inaccuracies. The question then is always "are these inaccuracies material ". I'm actually very impressed by TFA and it seems to me that if we get another order of magnitude improvement, it'll be around the accuracy of human accountants.
Yes but you have: 1. specific explicit training and certifications 2. someone to yell at and who can be fired for non-performance
Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…
It gets it right for me... https://chatgpt.com/share/687e8c28-7714-800c-abf4-e9cd3ce87b...
> But they do make categorization mistakes, which is a common source of errors. > Claude misclassifies a hosting cost (which counts as COGS) as a software subscription. This is simply asking too much of the agent. Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them! > What's Vercel? >> That's a hosting service. > Ah, so it goes to Cost of Goods Sold? >> Yeah…
> Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them! Your accountant as a 3rd party might have this issue. Your accountant that you hire as an employee to help you run your business is the one who should be doing this.
I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…
Lawyers have used it to write briefs; I would be very surprised if someone, somewhere wasn't slowly running a company into the ground by using ChatGPT or another LLM for accounting.
I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…
I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.
Most businesses don’t want to misrepresent their books, irrespective of the existence of shady accountants.
> But they do make categorization mistakes, which is a common source of errors. > Claude misclassifies a hosting cost (which counts as COGS) as a software subscription. This is simply asking too much of the agent. Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them! > What's Vercel? >> That's a hosting service. > Ah, so it goes to Cost of Goods Sold? >> Yeah…
Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
Regarding the diminishing returns with frontier models:
My general experience working with LLMs is that they perform better incrementally and to avoid contiguous-greedy approaches. Aggregate as you go and don't take on incrementally larger tasks. Keep the workload minimal.
Regarding agentic tool building: feels like I'm looking at a window into the future.