Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AccountingBench: Evaluating LLMs on real long-horizon business tasks

accounting.penrose.com

Post reply on HN
Print view View original

21–30 of 154 posts

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#21

Post by pavel_lishin »

vlade11115 wrote: post #5

I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…

Lawyers have used it to write briefs; I would be very surprised if someone, somewhere wasn't slowly running a company into the ground by using ChatGPT or another LLM for accounting.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#22

Post by falcor84 »

vlade11115 wrote: post #5

I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…

I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#23

Post by arm32 »

vachina wrote: post #11

An LLM is like a jackhammer, it works very well when you hold it tightly. If you let it loose it will sort of work for a while then it starts destroying everything around it.

Not sure if this is a good analogy. You're supposed to use a jackhammer with a very light grip.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#25

Post by o11c »

falcor84 wrote: post #22
vlade11115 wrote: post #5

I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…

I'm sure that any accounting trick that an LLM can think of is something that is also used by some shady human accountants. The proper response should not be to avoid/prohibit AI but to improve the validation mechanisms.

Counterpoint: if you detect a human accountant doing this, you can take action against the human. Computers will never meaningfully take the blame, and unfortunately usually mean not blaming any human either.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#26

Post by bigfishrunning »

arm32 wrote: post #23
vachina wrote: post #11

An LLM is like a jackhammer, it works very well when you hold it tightly. If you let it loose it will sort of work for a while then it starts destroying everything around it.

Not sure if this is a good analogy. You're supposed to use a jackhammer with a very light grip.

They have much better jackhammer metaphors over on JackerNews

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#27

Post by pton_xd »

Reading through the LLM log entries, it's just astounding the amount of depth current models are capable of. It's almost hard to comprehend that this is even possible. Yeah the current ones mess up after a while, but ... the future is going to be very interesting.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#28

Post by modeless »

pton_xd wrote: post #27

Reading through the LLM log entries, it's just astounding the amount of depth current models are capable of. It's almost hard to comprehend that this is even possible. Yeah the current ones mess up after a while, but ... the future is going to be very interesting.

Models that can think coherently for hours to solve IMO problems are likely going to do much better at this as well.

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#29

Post by School-Cotton »

Havoc wrote: post #18

Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…

It gets it right for me... https://chatgpt.com/share/687e8c28-7714-800c-abf4-e9cd3ce87b...

Re: AccountingBench: Evaluating LLMs on real long-horizon business tasks

#30

Post by crthpl »

Havoc wrote: post #18

Remember that test where you ask a LLM whether 9.11 or 9.9 is the bigger number? [Just checked gpt-4o still gets it wrong] I don't think you'll find many sane CFOs willing to send the resulting numbers to the IRS based on that. That's just asking to get nailed for tax fraud. It is coming for the very bottom end of bookkeeping work quite soon though, especially for first draft. There are a lot of people doing stuff li…

GPT-4o is so far behind the frontier; you shouldn't use it as an indicator of what LLMs are capable of.
Post reply on HN