Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM 5.2 is nearly as accurate as a human book keeper

toot-books.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: GLM 5.2 is nearly as accurate as a human book keeper

#91

Post by dylan604 »

walrus01 wrote: post #44
Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a n…

Sounds like Enron was just a bit early. They could have just blamed the AI system instead of becoming a meme with "Enron math"

Re: GLM 5.2 is nearly as accurate as a human book keeper

#92

Post by petercooper »

I wouldn't be surprised if it were more accurate based on the errors I've seen. I always eyeball the books and was confused when a £15k building popped up on our asset sheet. It turns out a "workshop" had been categorised as a building we had purchased, rather than the training session it actually was.

This is the importance of having layers and multiple sets of eyes on things, though. Even if it had got past me, my accountant would have surely queried it at year end, but that could be true of an LLM mistake too.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#93

Post by elevate_ »

Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

[flagged]

Re: GLM 5.2 is nearly as accurate as a human book keeper

#94

Post by Pragmata »

ofjcihen wrote: post #90
ACCount37 wrote: post #58

Earlier quoted context omitted.

Do you think human accountants are deterministic? Get a large enough org and watch your accounting grow an error margin.

Ha even if this was true (it’s not) you’re basically saying “humans will make some mistakes so let’s throw caution to the wind” which is probably the worst application of AI that I’ve heard yet.

>Ha even if this was true (it’s not)

You haven't met many accountants i see.

Regardless that's not what he is saying.

If there is an acceptable margin of error for humans, we should be able to measure it for AI, and once AI is within those margins then it should be feasible to replace the human.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#95

Post by Shitty-kitty »

petesergeant wrote: post #81
Shitty-kitty wrote: post #52

Earlier quoted context omitted.

The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.

isn't that true for humans too?

Humans are more likely to make small mistakes but the internal consistency check is pretty good at catching large errors. On top of that, fudging numbers to make everything add up is not something humans do (not unintentionally at least)

Re: GLM 5.2 is nearly as accurate as a human book keeper

#96

Post by order-matters »

walrus01 wrote: post #44
Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a n…

i specialize in invoice related analytics and business processes. AI can be useful for data extraction and saves me some typing time. even so, i dont blindl trust it because it sometimes makes very reasonable mistakes because invoices, quotes, and POs are sometimes structured in very informal ways. people misuse the lines, sublines, totals, and other data fields. they are often technically incorrect but when you look at them you know what they mean. i am not sure how to hand that off to something that has plausible deniability to guess even if it doesnt know

sometimes details are just in notes at the bottom and are applied to selectively applied to line items. sometimes the charge doesnt exist anywhere on the bill but there is an understanding (due to a separate agreement) of additional charges to be paid as a result of the invoice.

taxes sometimes are or are not explicitly stated

tariffs sometimes are or are not explicitly stated

when things are not explicitly stated or line item'd, they will usually still appear in the invoice total. so you have item 1 - $500, item 2 $500. total: $1300

At the end of the day invoices are often part of an ongoing communication / conversation between two organizations and they are created with an assumption that a rational and reasonable human who is in the loop with that conversation is going to read it.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#97

Post by scheme271 »

benjiro29 wrote: post #57
zerobees wrote: post #26

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM. If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of ta…

> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. From my understanding, you are the person signing off on the paperwork that is submitted to the IRS. There is this cache 22 with taxes. You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to under…

I don't think that's actually the case. If you get an tax attorney to document your treatment and indicate that they believe it's legal (e.g. a tax opinion letter), then you're off the hook for penalties and criminal liability (assuming the tax attorney did everything properly). You still have to pay the difference though.

An accountant may not save you from financial penalties but I believe that they are liable for them so you can recover it from their insurance.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#98

Post by jimnotgym »

mediaman wrote: post #73
walrus01 wrote: post #44

Earlier quoted context omitted.

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a n…

You can fix this simply by using normal controls. That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver. Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is…

I run a team that includes people that do this kind of work using ocr software that matches invoices to po's. No AI needed. This is a solved problem. Why are there people involved? Because sometimes the invoice and po don't match. For instance, price on invoice is higher than po, refer to buyer. Buyer is sick, supplier puts you on hold, no parts for your factory, lose millions... Would you trust an AI to choose what to do next? This might get referred to me to resolve and make a decision, not just on the facts available, but on other facts I can discover, and years of experience. I might end up making an unauthorised payment, would you give an AI that power?

Re: GLM 5.2 is nearly as accurate as a human book keeper

#99

Post by specproc »

walrus01 wrote: post #44
Diogenesian wrote: post #31

This shouldn't be ignored in the discussion here: The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are p…

If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era. edit: There's already a n…

I think it would be absolutely insane to hand over a serious-sized company's books to an LLM.

As a small consultancy though, looking forward to my next filing, and having just moved to a new and better-specced jurisdiction, I'm sorely tempted to outsource to Claude.

I've had mixed experience with accountants in the past. No horror stories, but I often feel I'm not getting everything laid out clearly, and that I don't fully understand the process.

I've got plenty of reasons to dislike LLMs in my own work, but when dealing with well-scoped but professionally gatekept things like tax or property transactions, they're an absolute godsend.

Re: GLM 5.2 is nearly as accurate as a human book keeper

#100

Post by jimnotgym »

mediaman wrote: post #79
walrus01 wrote: post #74

Earlier quoted context omitted.

Indeed so, a fairly mundane RFP, RFQ, buyer, receiver, accounts payable process will stop a lot of problems. If an agent is inserted at some stage in the process with a clear path to make a ticket/escalate to a human if it sees something it doesn't understand, the risk isn't absurdly high , in my opinion. I've seen so many reports of humans with the authority/ability to execute an outgoing SWIFT transfer who've been…

I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI. With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought…

But the hard rules only work up to the point that there is an exception. See my other post above. Occasionally a very senior person (hopefully senior) has to approve a payment that is outside of the rules, because it is something the rules did not anticipate (and now they are hardcoded into software and can't be changed).
Post reply on HN