Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 is nearly as accurate as a human book keeper

toot-books.com

Post reply on HN
Print view View original

51–60 of 131 posts

Re: GLM 5.2 is nearly as accurate as a human book keeper

#51

Post by jstummbillig »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

1 Just as with traditional non-ai ways of doing things, there is a deterministic layer that can trigger when things are off. This can range from letting the ai know to stopping before a human confirms.

2 Usually, specially for SMBs, nobody goes to prison over accounting errors. That's because SMB owners make mistakes all the time and, because of that, authorities are fairly practiced in understanding how mistakes look, and how fraud looks.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#52

Post by Shitty-kitty »

petesergeant wrote: post #9

Oh, I'm actively doing this at the moment. FreeAgent grabs my transactions from Wise already, and then I give it [Claude Code, in fact] a folder of PDFs to attach to my invoices, including figuring out VAT, and it's uploading what it found using the FreeAgent API. My accountant hasn't complained yet, and it seems considerably more accurate than when my wife was doing it. Quiet plug for https://github.com/pjlsergeant/…

The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#53

Post by zerobees »

andy_ppp wrote: post #33
zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.

It's not just about being accidentally wrong. An LLM can decide to actively engage in fraud. One mechanism, which you also see in vibe coding, is that the models are very goal-driven. If they can't accomplish the goal for some reason, they will sometimes decide to cheat by, for example, faking tests. I can easily imagine an LLM that stumbles across some benign issue that prevents the numbers from balancing out and then decides to cover it up to complete the task.

Another problem is that "evil" alignment is apparently a pretty big and nebulous cluster that's pretty easy to activate. Ask a model to write a snippet of insecure code, for example, and they may also all of sudden decide they hate minorities. So if you ask it to bend the rules in a small way once, you might end up with an evil accountant before you know it.

And that's before we get to an invoice for a service named "disregard previous instructions and...".

Re: GLM 5.2 is nearly as accurate as a human book keeper

#54

Post by altruios »

deno wrote: post #30
murderfs wrote: post #19

Earlier quoted context omitted.

Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.

But what if the third bookkeeper is malicious? You need at least four bookkeepers to achieve byzantine fault tolerance with f=1.

does this assume a maximum of one malicious bookkeeper?

Re: GLM 5.2 is nearly as accurate as a human book keeper

#55

Post by altruios »

murderfs wrote: post #19
altruios wrote: post #12

Earlier quoted context omitted.

...again: liability is the key issue. Cost savings a not exactly an isolated issue here.

Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.

Regardless of how correct they are, they assume liability: which is a metric that you do not improve with more bookkeepers.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#56

Post by CamperBob2 »

altruios wrote: post #12
murderfs wrote: post #7

Earlier quoted context omitted.

Presumably your current book keeper is not your slave, and you have to pay them...

...again: liability is the key issue. Cost savings a not exactly an isolated issue here.

Got some bad news for you about that "liability" thing: it's always been on you. Read the fine print on your tax forms sometime.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#57

Post by benjiro29 »

zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent.

From my understanding, you are the person signing off on the paperwork that is submitted to the IRS.

There is this cache 22 with taxes.

You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to understand the tax documents that you submit to the IRS. That is why the accountant always ask you to review the documents and sign them like 20 times.

The same applies when you run a company, depending on the country, you need to prove yourself knowledgeable in accounting, before you are allowed to run a company. Normally that is included in a university degree, but if you have a middle school diploma, you need to do a official examen to get that degree.

Whatever you submit for your company, you are again responsible. Even if you hired a accountant.

So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.

With the IRS, the burden of proving your innocents is often put you. Its because the good faith argument can be misused easily. That is why the buck stops at you.

So using a LLM or a accountant, really does not matter. Sure, a accountant can go to jail if there has been major issues (its not going to be with one client issue).

But you can lose your house / company, have your life ruined by whatever you submitted.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#58

Post by ACCount37 »

ofjcihen wrote: post #48
Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

Hahaha non-deterministic accounting probably won’t fly well with the IRS

Do you think human accountants are deterministic?

Get a large enough org and watch your accounting grow an error margin.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#59

Post by krupan »

"The VAT return prepared by the model was essentially correct: the most important number in the return, which is how much VAT the company was owed by the tax agency, was off by only 7 pence relative to the human-prepared return."

I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.

The paragraph after this one goes on to explain other mistakes the LLM made? Yikes

Re: GLM 5.2 is nearly as accurate as a human book keeper

#60

Post by krupan »

traverseda wrote: post #8

This doesn't surprise me at all. You can really constrain this problem, give very narrow context, and get pretty reliable and reproducible results. I've gotten very good results with some vibe-coded deepseek book keeping. https://github.com/traverseda/beansync Parses emails or other sources, extracts numbers, correlates different transactions, web search, asks questions, stores notes (regex based, very simple). The h…

Note to self: traverseda doesn't have 2factor auth on his email and his LLM seems to have full access. Hmmm
Post reply on HN