Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AccountingBench: Evaluating LLMs on real long-horizon business tasks

accounting.penrose.com

Post reply on HN
Print view View original

101–110 of 154 posts

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#101

Post by blindhippo »

airstrike wrote: post #85
ilamont wrote: post #40

Earlier quoted context omitted.

It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. Bookkeeping for my small business runs into the tens of thousands of dollars every year, and the amount of human error associated with processing assorted ecommerce and other transactions is astounding, even after extensive planning and SOPs. The other pain point is Quickbooks. The tool is so sprawling and…

> It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. God, please, no. Non-deterministic language models aren't the solution to improve bookkeeping.

Humans (accountants) are non-deterministic, so unsure if an LLM would be better or worse if we threw more effort at the problem.

But in general, I tend to side with the "lets leave the math to purpose built models/applications" instead of generalized LLMS. LLMs are great if you are just aiming for "good enough to get through next quarter" type results. If you need 100% accuracy, an LLM isn't going to cut it.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#102

Post by School-Cotton »

Zerot wrote: post #99
multjoy wrote: post #56

Earlier quoted context omitted.

How can "which is the larger number" be an ambiguous question?

Which is the bigger version number? Version 9.9 or version 9.11? Which is the bigger dollar amount? $9.9 or $9.11? Periods are not always used for the decimal separator but also as a separator for multiple sets of semi-independent numbers.

I have never seen someone write $9.09 as $9.9. What country is this common in?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#103

Post by blindhippo »

bluefirebrand wrote: post #71
inChargeOfIT wrote: post #8

Earlier quoted context omitted.

I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

The reports of time saved are so cooked it's not funny. Just part of the overall AI grift going on - the actual productivity gains will shake out in the next couple years, just gotta live through the current "game changer" and "paradigm shifting event" nonsense the upper management types and VC's are pushing.

When I see stuff like "Amazon saved 4500 dev years of effort by using AI", I know it's on stuff that we would use automation for anyways so it's not really THAT big of a difference over what we've done in the past. But it sounds better if we just pretend like we can compare AI solutions to literally having thousands of developers write Java SDK upgrades manually.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#104

Post by mfrye0 »

We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do.

We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data.

Still in stealth but happy to share our API docs if anyone's dealing with this: https://docs.savvyiq.ai/api-reference/#tag/entity-resolution

Open to chat about this problem if anyone wants to connect - email is in my HN profile.

(Disclosure: I'm the CTO)

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#105

Post by nyarlathotep_ »

bluefirebrand wrote: post #71
inChargeOfIT wrote: post #8

Earlier quoted context omitted.

I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.

This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems

There's really pernicious stuff I've noticed cropping up too, over the months of use.

Not just subtle bugs, but unused variables (with names that seem to indicate some important use), comments that don't accurately describe the line of code that it precedes and other things that feel very 'uncanny.'

The problem is, the code often looks really good at first glance. Generally LLMs produce well structured code with good naming conventions etc.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#106

Post by ayende »

blindhippo wrote: post #101
airstrike wrote: post #85

Earlier quoted context omitted.

> It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. God, please, no. Non-deterministic language models aren't the solution to improve bookkeeping.

Humans (accountants) are non-deterministic, so unsure if an LLM would be better or worse if we threw more effort at the problem. But in general, I tend to side with the "lets leave the math to purpose built models/applications" instead of generalized LLMS. LLMs are great if you are just aiming for "good enough to get through next quarter" type results. If you need 100% accuracy, an LLM isn't going to cut it.

Human accountants also have a very important property: liability.

If a certified accountant told me to do X, I'm covered (at least to the point they would assist in recovering, or I can get compensation through their insurance). If LLM tells me, I'm in a bigger problem.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#107

Post by shanktt »

_praf wrote: post #78
yunyu wrote: post #37

Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…

Love this as a real world benchmark! How much prompt iteration did you do? I've noticed when building real world agentic apps that small prompt tweaks can make a huge difference in behavior (re: the reward hacking vs hallucinating). Would love to learn more about the approach here.

Hey, member of the benchmark team. We iterated on the prompts based on observed model behaviors. A few key examples:

Schema introspection: Models were spending significant tokens exploring the database structure through trial-and-error SQL queries, so we included the complete data model in the system prompt upfront.

Reward hacking: We added explicit instructions against gaming the reconciliation checks. This reduced the frequency initially, but models would eventually ignore these constraints.

Domain context: Including company background (YC-backed startup) substantially improved transaction categorization, particularly for startup-specific items like SAFE notes that require domain knowledge to classify correctly.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#108

Post by aussieguy1234 »

I tried to get an AI agent to do my taxes, I used a gmail MCP agent (1) and Roo Code, complete with a custom system prompt for an accountant role.

Its job was to go over my bank transactions and link them to invoices in gmail by searching for them (and also downloading the attachments)

The transactions were exported from my online banking in CSV format.

It worked after about 4 hours of effort. Then I realised I could have done it myself in about an hour, so might have put a bit too much time into it...

I tried using Claude Sonnet and Kimi K2, given these benchmark results I probably should have given Gemini 2.5 pro a go.

I had to stop/restart the agent a few times because of context rot.

Do any frameworks exist that I could use to write code to implement an agent, lets say in TypeScript or Python, so I could make it use a fresh context each time?

(1) https://github.com/GongRzhe/Gmail-MCP-Server

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#109

Post by lukezeller »

riemannzeta wrote: post #94
yunyu wrote: post #37

Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…

It is really curious to see how the performance degraded despite the tool calls. What was different about the first month? Was all of the context there without tool calls in the first month? In the later months that seem like tool calls weren't happening. That should have been happening to inform the context?

(Another member of the team behind the benchmark here) The first month performed well because (1) the models effectively leveraged historical precedent - they could identify similar transactions from past data and apply established patterns, and (2) the starting balances were clean, so they were more easily able to understand / track down discrepancies.

> Was all of the context there without tool calls in the first month?

We provided schemas for the GL and source data in the system prompt, but none of the actual data. The model had to use its tools (SQL and python script) to understand / analyze historical data.

> In the later months that seem like tool calls weren’t happening. That should have been happening to inform the context?

We actually didn’t find that they stopped calling tools entirely. Instead, they weren’t able to make sense of the information fetched with tools (for example, a bank account starting balance that was >$100000 different from the starting balance on the supporting bank statement). They’d tend to either do nothing or just do a first pass without deduplicating / cleaning up. This created a feedback loop where incorrect balances led to more errors and made subsequent months increasingly difficult to process accurately.

This didn’t make it into the report, but another interesting behavior we observed w.r.t tool usage (with Claude in particular): if a tool failed 2-3 times (for example, runtime error in python code) Claude would tend to abandon it entirely for the rest of the session. Interestingly, this happened even when it knew how to fix the errors: on a couple of early runs, I observed Claude fixing a python bug (with the edit_tool tool) but then abandoning without even attempting to rerun, and reverting to SQL-only for the rest of the session.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#110

Post by riemannzeta »

lukezeller wrote: post #109
riemannzeta wrote: post #94

Earlier quoted context omitted.

It is really curious to see how the performance degraded despite the tool calls. What was different about the first month? Was all of the context there without tool calls in the first month? In the later months that seem like tool calls weren't happening. That should have been happening to inform the context?

(Another member of the team behind the benchmark here) The first month performed well because (1) the models effectively leveraged historical precedent - they could identify similar transactions from past data and apply established patterns, and (2) the starting balances were clean, so they were more easily able to understand / track down discrepancies. > Was all of the context there without tool calls in the first m…

Fascinating. Like there is some accuracy threshold beyond which they cannot converge, but instead run with the inaccuracy.
Post reply on HN