AccountingBench: Evaluating LLMs on real long-horizon business tasks • Hacker News Board

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#91

Post by luckystarr » Mon, Jul 21, 2025, 10:25 PM UTC

airstrike wrote: post #85

ilamont wrote: post #40

Earlier quoted context omitted.

It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. Bookkeeping for my small business runs into the tens of thousands of dollars every year, and the amount of human error associated with processing assorted ecommerce and other transactions is astounding, even after extensive planning and SOPs. The other pain point is Quickbooks. The tool is so sprawling and…

> It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. God, please, no. Non-deterministic language models aren't the solution to improve bookkeeping.

Well I've seen worse bookkeepers. "You know, you approved of the budget, but where are our customers payments in the balance sheets? We can't find them!" - "Uhm..."

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#92

Post by oblio » Mon, Jul 21, 2025, 10:26 PM UTC

bluefirebrand wrote: post #71

inChargeOfIT wrote: post #8

Earlier quoted context omitted.

I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

I think people are doing one of several things to get value:

0. Use it for research and prototyping, aka throwaway stuff.

2. Use it for studying an existing, complex project. More or less read only or very limited writes.

3. Use it for simple stuff they don't care much about and can validate quickly and reasonably accurately, the standard examples are CLI scripts and GUI layouts.

4. Segment the area in which the LLM works very precisely. Small functions, small modules, ideally they add tests from another source.

5. Boilerplate.

There can be a lot of value in those areas.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#93

Post by neom » Mon, Jul 21, 2025, 10:30 PM UTC

andy99 wrote: post #88

neom wrote: post #84

Posts like this kinda-sorta grind my gears, like... I get it, but also... accounting, like many real world tasks, is fundamentally a chain of precise and constrained and auditable operations. Humans approach these tasks through structured processes... we use roles, and we have checkpoints precisely because complexity compounds quickly and becomes unmanageable if tackled as one giant block. Expecting a single AI model…

It's a useless benchmark if everyone aces it. If some models do better than others and none saturate it, then is has some value, no? Permitting comparison is the point.

I agree, hence the heavy couching, so to your point I'm def just ranting a bit, because: I just think it would be more valuable to see some kind of MoA, I guess what I'm talking about is a bit of a different measurement, thinking in terms of economic outlook and understanding where we are and what can be done. I suspect more of this will shape our ability to understand how frontier models will impact the economy. Maybe I should just do my own evaluation, heh. :)

Edit: although to argue against myself, I suppose once a model can one-shot this stuff, my MoA comments become moot.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#94

Post by riemannzeta » Mon, Jul 21, 2025, 10:33 PM UTC

yunyu wrote: post #37

Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…

It is really curious to see how the performance degraded despite the tool calls. What was different about the first month? Was all of the context there without tool calls in the first month? In the later months that seem like tool calls weren't happening. That should have been happening to inform the context?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#95

Post by lordnacho » Mon, Jul 21, 2025, 11:02 PM UTC

Interestingly, one of my two big observations of LLM failure was also on an accounting task.

I thought it would be easy to do this, which is why I was surprised:

I had a folder full of bills, each of them with the VAT amount. Some were pictures, and some were PDFs. I asked for the total VAT for all 19 bills.

It took an immense number of prompts to get it to find the numbers correctly. It would get confused about reading the images as binary, that kind of thing. Or it would forget that it had to continue once it had found a few numbers. I got a total out in the end, but it took far too many prompts.

This is the only time I've come across a task a child could do that LLM failed at.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#96

Post by SpaceNoodled » Mon, Jul 21, 2025, 11:04 PM UTC

oblio wrote: post #92

bluefirebrand wrote: post #71

Earlier quoted context omitted.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

I think people are doing one of several things to get value: 0. Use it for research and prototyping, aka throwaway stuff. 2. Use it for studying an existing, complex project. More or less read only or very limited writes. 3. Use it for simple stuff they don't care much about and can validate quickly and reasonably accurately, the standard examples are CLI scripts and GUI layouts. 4. Segment the area in which the LLM…

What about 1. ?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#97

Post by theodorewiles » Mon, Jul 21, 2025, 11:26 PM UTC

For me this benchmark suggests that an LLM will try to “force the issue” which results in compounding errors. But I think the logical counterpoint is that you may be asking the LLM to come up an answer without all of the necessary details? Some of these are “baked into” historical transactions which is why it does well in months 1-2.

My takeaway is scaling in the enterprise is about making implicit information explicit.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#98

Post by pharrington » Mon, Jul 21, 2025, 11:36 PM UTC

lufenialif2 wrote: post #4

I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.

"a better auto complete" than what, specifically?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#99

Post by Zerot » Mon, Jul 21, 2025, 11:48 PM UTC

multjoy wrote: post #56

ASpring wrote: post #33

Earlier quoted context omitted.

> Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] Interesting, 4o got this right for me in a couple different framings including the simple "Which number is larger, 9.9 or 9.11?". To be a full apologist, there are a few different places (a lot of software versioning as one) where 9.11 is essentially the bigger number so it may be an ambiguous…

How can "which is the larger number" be an ambiguous question?

Which is the bigger version number? Version 9.9 or version 9.11? Which is the bigger dollar amount? $9.9 or $9.11?

Periods are not always used for the decimal separator but also as a separator for multiple sets of semi-independent numbers.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#100

Post by ryeguy_24 » Tue, Jul 22, 2025, 12:01 AM UTC

Isn’t there a whole bunch of dependency here related to prompting and methodology that would significantly impact overall performance? My gut instinct is that there are many many ways to architect this around the LLMs and each might yield different levels of accuracy. What do others think?

Edit: In reading more, I guess this is meant to be a dumb benchmark to monitor through time. Maybe that’s the aim here instead of viability as an auto close tool.