AccountingBench: Evaluating LLMs on real long-horizon business tasks
accounting.penrose.com
AccountingBench: Evaluating LLMs on real long-horizon business tasks
1–10 of 154 posts
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#2not a game on Steam? :(
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#3We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome
A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the books are off by 20% or when a bridge is 5 inches too short to reach the other side
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#4I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#5I love the site design.
> There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated ones to make the numbers add up.
This is hilarious. I wonder if someone is unintentionally committing fraud by blindly trusting LLMs with accounting. Or even worse, I bet that some governments are already trying to use LLMs to make accounting validators. My government sure wants to shove LLMs into digital government services.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#6I find the same issues (though with much lower stakes) when using an LLM to determine the outcome of a turn in a game. I'm working on something called "A Trolly (problem) Through Time" where each turn is a decade starting with the 1850s, and you are presented with historic figures on a train track, and you have to chose whether to actively spare the person on your track for a potential unknown figure on the other side, or let the train run them over.
It works well as a narrative, but the second I started adding things like tracking high level macro effects of the decisions, within a couple of turns the world's "Turmoil" goes from 4/10 to a 10/10... even when the person that was killed would have been killed IRL.
Sonnet 4, o4-mini, and GPT 4o-mini all had the same world ending outcomes not matter who you kill. Killing Hitler in 1930s: 10/10 turmoil, Killing Lincoln in the 1850s: 10/10 turmoil in the first turn.
I've come to the realization, the LLM shouldn't be used for the logic, and instead needs to be used to just narrate the choices you make.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#7> In fact, we explicitly prompt against this behavior in no uncertain terms, but the instructions – and the entire spirit of the task – are lost in the interest of making forward progress
LLMs and humans are quite alike. :) I notice that a few models will give up instead of ignoring their instructions and that's the model I would want working on tasks like this. An LLM should be able to categorize and reconcile transactions, but if it's not sure, it should quit and give it back to the humans.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#8I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.
I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#9not a game on Steam? :(
If you want to treat yourself with an accounting game night, there's this one built by @patio11: https://keshikomisimulator.com/
Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks
#10I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.
I feel it does essentially save a lot of time in bookkeeping, but doesn’t negate the need for a human bookkeeper. Who knows what they’re doing