121–130 of 154 posts
I love the site design. > There's an obvious question looming here — if the models got so confused, how did they consistently pass the reconciliation checks we described above? It may seem like the ability to make forward progress is a good proxy for task understanding and skill, but this isn't necessarily the case. There are ways to hack the validation check – inventing false transactions or pulling in unrelated one…
> In fact, we explicitly prompt against this behavior in no uncertain terms, but the instructions – and the entire spirit of the task – are lost in the interest of making forward progress LLMs and humans are quite alike. :) I notice that a few models will give up instead of ignoring their instructions and that's the model I would want working on tasks like this. An LLM should be able to categorize and reconcile trans…
> but if it's not sure, it should quit Can it be sure or not? I've never been able to get LLMs to give confidence measures that match their actual outputs. I'll ask an LLM "Are you sure?" and it'll reply "Absolutely" when it's output is completely wrong, or it'll backtrack on a correct output with "I should not have provided an answer when I was unsure. Here is an answer I am sure of..." and then provide something co…
> Claude did you just try to completely remove my large file *BEFORE* checking it into git LFS?
> You're absolutely right! I should not have attempted an 'rm' comment on unstaged data. I guess I got a little frustrated with git, haha!Earlier quoted context omitted.
Human accountants also have a very important property: liability. If a certified accountant told me to do X, I'm covered (at least to the point they would assist in recovering, or I can get compensation through their insurance). If LLM tells me, I'm in a bigger problem.
Most small businesses cannot afford CPAs for everyday tasks. At best a CPA signs off on the annual summaries. Most day to day work is done by bookkeepers who are not CPAs. In my area (Vermont) the going rate for a good CPA is $200/hr. Bookkeepers are $20-30/hr.
I sent this to accounting friends and this aligns with what I've been going through trying to use LLMs to create a game from scratch. Seems like the current best use case for language models (even with agent mode) is to feed it exactly what you want to get out, essentially turning it into a better auto complete. Still saves tons of time, but it isn't a panacea.
they have emergent capabilities, like "translating" instructions/questions in X to the probable answers in Y, but i think people are getting way way ahead of themselves with those. these things still fundamentally cant think, and we can try to mimic thinking with scaffolding but then your just going to learn the bitter lesson again
Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
Eg I can imagine invariants like balancing anccounts are essentially mechanical, but classifying spending categories currently requires judgement (and therefore human-curated ground-truth). But I’m curious if there are approaches to reduce the latter, say with constructing a semantic graph ontology for the domain or something along those lines.
I guess there is an interesting duality here in that if you solve the eval you have also created a valuable business!
Earlier quoted context omitted.
I'm not even sure it saves a ton of time to be honest. It sure _feels_ like I spend more time writing up tasks and researching/debugging hallucinations than just doing the thing myself.
This is consistently my experience too, I'm seriously just baffled by reports of time saved. I think it costs me more time cleaning up its mistakes than it saves me by solving my problems
essentially, a longer horizon increases chances of mistakes, increasing time needed to find and fix them. so at one point that becomes greater than the time saved in not having to do it myself
this is why im not bullish on AI agents. task horizon is too long and dynamical
I guess having access to tools / running Python would make all the difference.
"Available Tools: [...] create_tool(tool_name, description, python_code, parameters) Create a new tool that can execute Python code. The tool becomes immediately available for use. Tools can call other tools and return different formats based on context (formatted for direct calls, raw data for tool-to-tool calls)."
We've been on this train of not caring about the details for so long but AI just amps it up. Non-deterministic software working on things that have extremely precise requirements is going to have a bad outcome A company may be OK with an AI chatbot being so bad it results in 5-20% of customers getting pissed off and not having a 5-star experience. The SEC and DOJ (and shareholders) are not going to be happy when the…
Human accountants are notoriously non-deterministic too, and any sufficiently complex accounting process contains inaccuracies. The question then is always "are these inaccuracies material ". I'm actually very impressed by TFA and it seems to me that if we get another order of magnitude improvement, it'll be around the accuracy of human accountants.
Hey all, member of the benchmark team here! The goal for this project was to see how LLMs well could do bookkeeping without an overly opinionated scaffold. We gave them access to processed transaction records and code execution tools, but it was up to them to choose exactly how to use those. Claude and Grok 4 did reasonably well (within CPA baselines) for the first few months, but tended to degrade as more data came…
Earlier quoted context omitted.
How can "which is the larger number" be an ambiguous question?
Which is the bigger version number? Version 9.9 or version 9.11? Which is the bigger dollar amount? $9.9 or $9.11? Periods are not always used for the decimal separator but also as a separator for multiple sets of semi-independent numbers.
also $9.9 is clearly 9 dollars and 90 cents.