Earlier quoted context omitted.
Not sure if this is a good analogy. You're supposed to use a jackhammer with a very light grip.
They have much better jackhammer metaphors over on JackerNews
yes but i cant open that at work
AccountingBench: Evaluating LLMs on real long-horizon business tasks
131–140 of 154 posts
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#132An LLM is like a jackhammer, it works very well when you hold it tightly. If you let it loose it will sort of work for a while then it starts destroying everything around it.
and investors are frothing at the mouth to put a jackhammer in every home
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#133But can't it, literally, hallucinate raw data at any point in the run?
they did mention that it would make up fake transactions to balance the book
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#134We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do. We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data. Still in stealth but happy to share our API docs if anyone…
entity resolution is the killer feature. context engineering is the problem with this benchmark attempt. The agent plan seemed to one shot, and the fact that the LLMs could write their own tools without validation or specific multi shot examples is worrisome. To me way to much left to the whims of the llms - with out proper context.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#135Love the old school microsoft interface. Feels familiar sight when my system is failing.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#136Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
Do you have any plan to open source the benchmark in the future?
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#137Earlier quoted context omitted.
I think people are doing one of several things to get value: 0. Use it for research and prototyping, aka throwaway stuff. 2. Use it for studying an existing, complex project. More or less read only or very limited writes. 3. Use it for simple stuff they don't care much about and can validate quickly and reasonably accurately, the standard examples are CLI scripts and GUI layouts. 4. Segment the area in which the LLM…
What about 1. ?
7 8 1 :-p
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#138Earlier quoted context omitted.
It's a start. The world needs a better way to handle bookkeeping, and the existing tools sure aren't cutting it. Bookkeeping for my small business runs into the tens of thousands of dollars every year, and the amount of human error associated with processing assorted ecommerce and other transactions is astounding, even after extensive planning and SOPs. The other pain point is Quickbooks. The tool is so sprawling and…
How small is your small business? My book keeping expenses are $120 a year, the cost of excellent saas software. I’ve found double entry books one of the most beautifully simple, yet powerful ideas I’ve ever come across. It’s hard to imagine how a balance sheet could be improved or disrupted. The balance sheet for my small business is the same as Apple and Alphabet’s and that still blows my mind.
I wonder if parent post is alluding to the number of hours spent bookkeeping? As a percentage of somebody's time, I could see that getting reasonably expensive.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#139[deleted]
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#140Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
> We conducted three runs per experiment and selected the run with the highest final accuracy for inclusion in the chart (though illustrative examples and anecdotes may be drawn from any of the runs).
Can you comment on the variance? It's impressive that models are able to do this consistently with 100% accuracy in the early months, but it would be less so if there was any significant degree of variance amongst the three runs (e.g. 90%, 95%, 100%.)