Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AccountingBench: Evaluating LLMs on real long-horizon business tasks

accounting.penrose.com

Post reply on HN
Print view View original

141–150 of 154 posts

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#141

Post by LgLasagnaModel »

lordnacho wrote: post #95

Interestingly, one of my two big observations of LLM failure was also on an accounting task. I thought it would be easy to do this, which is why I was surprised: I had a folder full of bills, each of them with the VAT amount. Some were pictures, and some were PDFs. I asked for the total VAT for all 19 bills. It took an immense number of prompts to get it to find the numbers correctly. It would get confused about read…

“ This is the only time I've come across a task a child could do that LLM failed at.”

Consider yourself lucky. It’s the people who haven’t run into something like this that will end up placing too much trust in these tools.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#142

Post by mrbungie »

luckystarr wrote: post #91
airstrike wrote: post #85

Earlier quoted context omitted.

> It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. God, please, no. Non-deterministic language models aren't the solution to improve bookkeeping.

Well I've seen worse bookkeepers. "You know, you approved of the budget, but where are our customers payments in the balance sheets? We can't find them!" - "Uhm..."

Sure, let's compare an almost global failure mode with the worst examples of knowledge workers. ffs.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#143

Post by anentropic »

lukezeller wrote: post #109
riemannzeta wrote: post #94

Earlier quoted context omitted.

It is really curious to see how the performance degraded despite the tool calls. What was different about the first month? Was all of the context there without tool calls in the first month? In the later months that seem like tool calls weren't happening. That should have been happening to inform the context?

(Another member of the team behind the benchmark here) The first month performed well because (1) the models effectively leveraged historical precedent - they could identify similar transactions from past data and apply established patterns, and (2) the starting balances were clean, so they were more easily able to understand / track down discrepancies. > Was all of the context there without tool calls in the first m…

sounds like it might work better to do each month as a fresh session, instead of iterating within one session and accumulating lots of context?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#144

Post by yunyu »

anentropic wrote: post #143
lukezeller wrote: post #109

Earlier quoted context omitted.

(Another member of the team behind the benchmark here) The first month performed well because (1) the models effectively leveraged historical precedent - they could identify similar transactions from past data and apply established patterns, and (2) the starting balances were clean, so they were more easily able to understand / track down discrepancies. > Was all of the context there without tool calls in the first m…

sounds like it might work better to do each month as a fresh session, instead of iterating within one session and accumulating lots of context?

To be clear, this is what we ended up doing

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#145

Post by yunyu »

mfrye0 wrote: post #104

We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do. We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data. Still in stealth but happy to share our API docs if anyone…

We solved this at Ramp on the expenses/AP side with an agentic RAG implementation and a custom embedding model, backed by D&B/Google/user-submitted corrections.

If curious, details here:

https://engineering.ramp.com/post/transaction-embeddings

https://engineering.ramp.com/post/fixing-merchant-classifica...

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#146

Post by scrollaway »

mrbungie wrote: post #142
luckystarr wrote: post #91

Earlier quoted context omitted.

Well I've seen worse bookkeepers. "You know, you approved of the budget, but where are our customers payments in the balance sheets? We can't find them!" - "Uhm..."

Sure, let's compare an almost global failure mode with the worst examples of knowledge workers. ffs.

I’m coming back from my accountant right now. He uses winbooks and has interns doing the books. I have no tooling to do it for him, and I’m seeing absurdities such as a 4000 usd refund being processed as cashback, and simple typos not being caught. I wish I was working with an AI instead of this nonsense. It’d be far more accurate.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#147

Post by bluefirebrand »

samrus wrote: post #126
bluefirebrand wrote: post #71

Earlier quoted context omitted.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

ive found that the shorter the "task horizon" the more time saved essentially, a longer horizon increases chances of mistakes, increasing time needed to find and fix them. so at one point that becomes greater than the time saved in not having to do it myself this is why im not bullish on AI agents. task horizon is too long and dynamical

So here's my problem, ultimately

If the task horizon for the LLM is shorter than writing it yourself, this likely means that the task is well defined and has an easy to access answer

For this type of common, well defined task we shouldn't be comparing "how long it takes for the LLM" against "how long it takes to write"

We should be comparing against "how long it takes to find the right answer on SO"

If you use this metric, I bet you the best SO answer, which is also likely the first google result, is just as fast as the LLM. Maybe faster

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#148

Post by mfrye0 »

yunyu wrote: post #145
mfrye0 wrote: post #104

We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do. We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data. Still in stealth but happy to share our API docs if anyone…

We solved this at Ramp on the expenses/AP side with an agentic RAG implementation and a custom embedding model, backed by D&B/Google/user-submitted corrections. If curious, details here: https://engineering.ramp.com/post/transaction-embeddings https://engineering.ramp.com/post/fixing-merchant-classifica...

Very cool. I read through those links - really sophisticated setup. We're experimenting with something similar on the embeddings side.

Having dealt with this challenge at my last 3 companies, it's easy to hack together something that works most of the time. The hard part is dealing with gnarly customer inputs, the long tail of private businesses globally, and getting close to 100% accuracy (important for legal and risk use cases).

We're building what's essentially an AI-powered version of D&B - combining government registrar data globally with real-time web data at scale. Much more accurate on obscure entities and way faster updates than the legacy providers.

I actually shot you an email - would love to chat more about this if you're up for it.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#149

Post by mfrye0 »

DrStartup wrote: post #134
mfrye0 wrote: post #104

We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do. We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data. Still in stealth but happy to share our API docs if anyone…

entity resolution is the killer feature. context engineering is the problem with this benchmark attempt. The agent plan seemed to one shot, and the fact that the LLMs could write their own tools without validation or specific multi shot examples is worrisome. To me way to much left to the whims of the llms - with out proper context.

Yes, none of the top LLMs can do entity resolution well yet. I constantly see them conflate entities with similar names - they'll confidently cite 3 sources about what appears to be one company, but the sources are actually about 3 different businesses with similar names.

The fundamental issue is that LLMs don't have a concept of canonical entity identity. They pattern match on text similarity rather than understanding that "Apple Inc" and "Apple Records" are completely different entities. It gets even worse when you realize companies can legally have identical names in the same country - text matching becomes completely unreliable.

Without proper entity grounding, any business logic built on top becomes unreliable.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#150

Post by jjmarr »

asgraham wrote: post #120
lucianbr wrote: post #43

Earlier quoted context omitted.

You're saying P=NP, I think.

Not to agree with GP, but I think it’s more accurate to say they’re saying “if validation is quick (to code), who cares how long a solution takes an AI because computation is cheap.” They’re not really making any claims about how quickly the AI can solve relative to the validation, which is what P vs NP is about.

This.

Yes, Cline generates a bunch of junk very quickly and makes dumb mistakes. If I can filter out those dumb mistakes automatically, I don't have to supervise it. It might burn 20¢ on fixing a mistake no human developer would make, but I don't have to pay a human developer.

Post reply on HN