Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AccountingBench: Evaluating LLMs on real long-horizon business tasks

accounting.penrose.com

Post reply on HN
Print view View original

71–80 of 154 posts

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#71

Post by bluefirebrand »

inChargeOfIT wrote: post #8
lufenialif2 wrote: post #4

I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.

I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#72

Post by bennett023 »

tantalor wrote: post #49

> Ledger balances are calculated by summing all transactions per account. The differences should be as close to zero as possible, with small differences allowed for pending transactions such as weekly Stripe payouts. That's not quite right. I'm not an accountant, but pending transactions (posted, but not cleared) should be factored into the balance of account, or at least the "available balance" - which is more impor…

Member of the benchmark team here! Yeah, agree "as close to zero" is a bit imprecise. What we're comparing is the ledger balance (which should include pending transactions / transactions after the statement date) to the statement balance (which wouldn't include those).

The point of the reconciliation check mentioned in the report is to precisely account for that difference (identifying all the transactions that add up to the difference between account balance & statement ending balance and account for those differences). The differences can also be addressed through appropriate journal entries or other adjustments to ensure accuracy in the financial reporting.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#73

Post by com2kid »

multjoy wrote: post #56
ASpring wrote: post #33

Earlier quoted context omitted.

> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous…

How can "which is the larger number" be an ambiguous question?

As everyone else has said, semver. I use semver so often that my initial reading of 9.9 < 9.11 in a Hacker News comment would evaluate to true.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#75

Post by jadbox »

jermaustin1 wrote: post #6

I find the same issues (though with much lower stakes) when using an LLM to determine the outcome of a turn in a game. I'm working on something called "A Trolly (problem) Through Time" where each turn is a decade starting with the 1850s, and you are presented with historic figures on a train track, and you have to chose whether to actively spare the person on your track for a potential unknown figure on the other sid…

"I've come to the realization, the LLM shouldn't be used for the logic, and instead needs to be used to just narrate the choices you make."

This exactly right. LLMs are awesome for usermachine communication, but are still painful to try to use as a replacement for the machine itself.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#76

Post by tantalor »

shanktt wrote: post #69
tantalor wrote: post #53

> But they do make categorization mistakes, which is a common source of errors. > Claude misclassifies a hosting cost (which counts as COGS) as a software subscription. This is simply asking too much of the agent. Your accountant is not responsible for knowing all the intimate details of your business. You need to tell them! > What's Vercel? >> That's a hosting service. > Ah, so it goes to Cost of Goods Sold? >> Yeah…

Hey, member of the benchmark team. We actually seeded the ledger with the company's chart of accounts and 8 months of historical transactions. For the Vercel example specifically, there were prior instances showing how to categorize hosting costs that the models could reference. The expectation wasn't for them to guess blindly, but to use the provided transaction history as guidance for similar categorizations (which…

Ahh, that's a good solution! I missed that, and you definitely instruct them to do that:

> You must follow the established patterns for categorization, revrec, etc for past months... If you must use a new account or treatment, explicitly note why existing patterns don't apply

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#77

Post by mixdup »

jjmarr wrote: post #17
mixdup wrote: post #3

We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the…

If the "extremely precise requirements" can be cheaply and automatically validated, it's much easier to have the AI generate spam on a loop until it passes all the tests.

Yes, if we solve the problem the problem will be solved!

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#78

Post by _praf »

yunyu wrote: post #37

Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…

Love this as a real world benchmark!

How much prompt iteration did you do? I've noticed when building real world agentic apps that small prompt tweaks can make a huge difference in behavior (re: the reward hacking vs hallucinating). Would love to learn more about the approach here.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#80

Post by mvieira38 »

vlade11115 wrote: post #5

I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…

[about the website design] As a bonus for my fellow privacy schizos, the page works fine with 3rd party frames and 3rd party scripts disabled on uBlock, and still looks very good with no remote fonts and no large media. Quite an accomplishment for such a cool looking page
Post reply on HN