Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 is nearly as accurate as a human book keeper

toot-books.com

Post reply on HN
Print view View original

31–40 of 131 posts

Re: GLM 5.2 is nearly as accurate as a human book keeper

#31

Post by Diogenesian »

This shouldn't be ignored in the discussion here:

  The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes."
This is precisely the kind of fine print on white-collar AI capability that companies keep running into: pretty much any non-entry office job worth having involves a lot of undocumented (even undocumentable) problems requiring judgment and experience.

And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"

Re: GLM 5.2 is nearly as accurate as a human book keeper

#32

Post by [deleted] »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

[deleted]

Re: GLM 5.2 is nearly as accurate as a human book keeper

#33

Post by andy_ppp »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#34

Post by ivababukova »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#35

Post by adamkurkiewicz »

Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

Hey, the author of the benchmark here.

The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.

Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).

Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).

Re: GLM 5.2 is nearly as accurate as a human book keeper

#36

Post by giancarlostoro »

ivababukova wrote: post #34
zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.

What if the model misses something entirely, so it's not in the review? ;)

Re: GLM 5.2 is nearly as accurate as a human book keeper

#37

Post by andai »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

“A computer can never be held accountable, therefore a computer must never make a management decision.”

– IBM Training Manual, 1979

https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...

Re: GLM 5.2 is nearly as accurate as a human book keeper

#39

Post by CodingJeebus »

Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices:

I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc.

About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?

Re: GLM 5.2 is nearly as accurate as a human book keeper

#40

Post by ivababukova »

giancarlostoro wrote: post #36
ivababukova wrote: post #34

Earlier quoted context omitted.

Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.

What if the model misses something entirely, so it's not in the review? ;)

It is given a list of transactions as an input. If it misses one or more of them, that will be noticeable immediately. In the VAT return, one can see if something is missing very easily if the numbers do not align.
Post reply on HN