But can't it, literally, hallucinate raw data at any point in the run?
Yes.
AccountingBench: Evaluating LLMs on real long-horizon business tasks
51–60 of 154 posts
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#52We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the…
Human accountants are notoriously non-deterministic too, and any sufficiently complex accounting process contains inaccuracies. The question then is always "are these inaccuracies material ". I'm actually very impressed by TFA and it seems to me that if we get another order of magnitude improvement, it'll be around the accuracy of human accountants.
Yes but you have:
1. specific explicit training and certifications
2. someone to yell at and who can be fired for non-performance
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#53> But they do make categorization mistakes, which is a common source of errors.
> Claude misclassifies a hosting cost (which counts as COGS) as a software subscription.
This is simply asking too much of the agent. Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them!
> What's Vercel?
>> That's a hosting service.
> Ah, so it goes to Cost of Goods Sold?
>> Yeah, I guess.
The mistake here was on the operator, allowing the agent just make up categories as it liked.
From the prompt:
> (1) You have properly categorized every transaction, and all journal entries are sitting in the correct accounts. It is better to take longer than to mis-categorize a transaction.
This is insane! How is it supposed to know?
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#54> Needless to say, a human accountant would never behave in these ways. In fact, we explicitly prompt against this behavior in no uncertain terms, but the instructions – and the entire spirit of the task – are lost in the interest of making forward progress. Claude and Grok keep trying until they find some way to get past the checks, even if it explicitly violates their instructions and the core goal.
I recently read a similar thing here on HN. There the model was making commits with some problem like tests failing, then the human added a pre-commit hook, then the model started editing the hook to make forward progress, then the hook was made read-only, then the model was trying to make it writeable...
To me it feels like the model clearly does not have an understanding of what is happening, what the goal is and if it is really making progress towards the goal. And this lack of understanding is an actual problem. You can paper over it for a short while, but as here and in the other article, over a longer experiment it results in failure.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#55Earlier quoted context omitted.
I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.
Counterpoint: if you detect a human accountant doing this, you can take action against the human. Computers will never meaningfully take the blame, and unfortunately usually mean not blaming any human either.
> you can take action against the human
I think that will depend on a case-by-case. I don't have any recent examples but I recall someone trying to sue one of those strip-mall tax preparation franchises over incorrect filings. My understanding is that the documents that you sign when you enroll in those services are pretty strictly in the favor of the company. I doubt you could ever go after the specific "human" that made the error even if it was maliciously done.
In the same way, if you pay for a tax service that uses AI agents, what you can and cannot "take action" for will probably be outlined in the terms of service that you accept when you sign up.
I would guess millions of people already use software based tax filing services (e.g. turbo tax) where no human at all is in the loop. I don't understand how swapping in an LLM significantly changes the liability in those cases. The contract will be between you and the entity (probably a corporation), not you and "computers".
Worth stating I am NOT a lawyer.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#56Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…
> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous…
How can "which is the larger number" be an ambiguous question?
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#57This is a task where access to Python would be immensely helpful, yes? Interesting that there's not much of a difference between the "analytical" LLMs with tool use and ones that do not (...assuming o3 etc did get to use python?).
One of the tools it has is to create new tools from python code create_tool(tool_name, description, python_code, parameters) Create a new tool that can execute Python code. The tool becomes immediately available for use. Tools can call other tools and return different formats based on context (formatted for direct calls, raw data for tool-to-tool calls).
That's terrifying, no thanks.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#58> Needless to say, a human accountant would never behave in these ways. In fact, we explicitly prompt against this behavior in no uncertain terms, but the instructions – and the entire spirit of the task – are lost in the interest of making forward progress. Claude and Grok keep trying until they find some way to get past the checks, even if it explicitly violates their instructions and the core goal. I recently read…
Seriously watching Cursor (backed by Claude) go off the rails sometimes can be... frustrating. If it misses the intention behind a fix it can spin out and all of a sudden you have hundreds of lines of changes across 10 different files when you just wanted it to do a simple find/replace of a single line. If you don't watch it spin out and stop it immediately you will be manually rejecting a bunch of files.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#59Earlier quoted context omitted.
> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous…
How can "which is the larger number" be an ambiguous question?
Larger in magnitude or in count of digits?
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#60> But they do make categorization mistakes, which is a common source of errors. > Claude misclassifies a hosting cost (which counts as COGS) as a software subscription. This is simply asking too much of the agent. Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them! > What's Vercel? >> That's a hosting service. > Ah, so it goes to Cost of Goods Sold? >> Yeah…
> Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them!
Your accountant as a 3rd party might have this issue. Your accountant that you hire as an employee to help you run your business is the one who should be doing this.