Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AccountingBench: Evaluating LLMs on real long-horizon business tasks

accounting.penrose.com

Post reply on HN
Print view View original

41–50 of 154 posts

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#41

Post by falcor84 »

o11c wrote: post #25
falcor84 wrote: post #22

Earlier quoted context omitted.

I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.

Counterpoint: if you detect a human accountant doing this, you can take action against the human. Computers will never meaningfully take the blame, and unfortunately usually mean not blaming any human either.

But still - if there's a way to detect accountants doing it - let's focus on making that detection even easier.

On a related note, can we use something like GAN here, with auditor AIs trained against accountant AIs?

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#42

Post by synalx »

jermaustin1 wrote: post #6

I find the same issues (though with much lower stakes) when using an LLM to determine the outcome of a turn in a game. I'm working on something called "A Trolly (problem) Through Time" where each turn is a decade starting with the 1850s, and you are presented with historic figures on a train track, and you have to chose whether to actively spare the person on your track for a potential unknown figure on the other sid…

I wonder if this is due to the common trope in science fiction literature that changing the past in even a small way has a butterfly effect of unintended and frequently disastrous consequences.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#43

Post by lucianbr »

jjmarr wrote: post #17
mixdup wrote: post #3

We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the…

If the "extremely precise requirements" can be cheaply and automatically validated, it's much easier to have the AI generate spam on a loop until it passes all the tests.

You're saying P=NP, I think.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#44

Post by [deleted] »

Havoc wrote: post #18

Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…

[deleted]

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#45

Post by Bjartr »

levocardia wrote: post #14

This is a task where access to Python would be immensely helpful, yes? Interesting that there's not much of a difference between the "analytical" LLMs with tool use and ones that do not (...assuming o3 etc did get to use python?).

One of the tools it has is to create new tools from python code

create_tool(tool_name, description, python_code, parameters)

Create a new tool that can execute Python code.

The tool becomes immediately available for use. Tools can call other tools and return different formats based on context (formatted for direct calls, raw data for tool-to-tool calls).

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#46

Post by falcor84 »

mixdup wrote: post #3

We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the…

Human accountants are notoriously non-deterministic too, and any sufficiently complex accounting process contains inaccuracies. The question then is always "are these inaccuracies material". I'm actually very impressed by TFA and it seems to me that if we get another order of magnitude improvement, it'll be around the accuracy of human accountants.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#47

Post by louthy »

bigfishrunning wrote: post #26
arm32 wrote: post #23

Earlier quoted context omitted.

Not sure if this is a good analogy. You're supposed to use a jackhammer with a very light grip.

They have much better jackhammer metaphors over on JackerNews

Bravo!

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#48

Post by herval »

arm32 wrote: post #23
vachina wrote: post #11

An LLM is like a jackhammer, it works very well when you hold it tightly. If you let it loose it will sort of work for a while then it starts destroying everything around it.

Not sure if this is a good analogy. You're supposed to use a jackhammer with a very light grip.

I think it actually holds truer to it working better with a _lighter grip_. LLMs tend to conclude the wrong thing if you over-control them (more context is what makes them less and less reliable over time, as in those demos), and trying to force a model to execute A+B+C=D in sequence is way harder than giving it a bunch of tools to arrive to conclusion D

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#49

Post by tantalor »

> Ledger balances are calculated by summing all transactions per account. The differences should be as close to zero as possible, with small differences allowed for pending transactions such as weekly Stripe payouts.

That's not quite right. I'm not an accountant, but pending transactions (posted, but not cleared) should be factored into the balance of account, or at least the "available balance" - which is more important the the "current balance".

The idea that you can "allow" accounting discrepancies as "those are probably pending" is wild.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#50

Post by mdaniel »

yoyohello13 wrote: post #32
School-Cotton wrote: post #29

Earlier quoted context omitted.

It gets it right for me... https://chatgpt.com/share/687e8c28-7714-800c-abf4-e9cd3ce87b...

Ah, wouldn’t be an LLM discussion thread without one of these “it works/doesn’t” conversations.

If it makes you feel any better, the other infamous one "I spend so much time chasing hallucinations, I could have done it myself" is currently a sibling comment
Post reply on HN