Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

321–330 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#321

Post by dragandj »

maccard wrote: post #136
ohgr wrote: post #107

Earlier quoted context omitted.

You missed the end of the supply chain. Paying users. Who magically disappear below market sustaining levels of sales when asked to pay.

I never said it was sustainable, and even if it was, OP asked for a business model. Customers don’t need a business model, they’re customers. The same is true for any non essential good or service.

Than any silly idea can be a business model. Suppose I collect dust from my attic and hope to sell it as an add-on on my neighbor's lemonade stand, with a hefty profit for the neighbor, who is getting paid by me $10 to add a handful of dust in each glass and sell it to the customers for $1. The neighbor accepts. It's a business model, at least until I don't run of existing funds or the last customer leaves in disguist. At which point exactly that silly idea stops being an unsustainable business model and becomes a silly idea? I guess at least as early as I see that the funds are running up, and I need to borrow larger an larger lumps of money each time to keep spinning the wheel...

Re: Recent AI model progress feels mostly like bullshit

#322

Post by bambax »

otabdeveloper4 wrote: post #319
AIPedant wrote: post #118

Earlier quoted context omitted.

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

Anecdotally: schoolkids are at the leading edge of LLM innovation, and nowadays all homework assignments are explicitly made to be LLM-proof. (Well, at least in my son's school. Yours might be different.) This effectively makes LLMs useless for education. (Also sours the next generation on LLMs in general, these things are extremely lame to the proverbial "kids these days".)

How do you make homework assignments LLM-proof? There may be a huge business opportunity if that actually works, because LLMs are destroying education at a rapid pace.

Re: Recent AI model progress feels mostly like bullshit

#323

Post by delian66 »

mlinhares wrote: post #224
aerhardt wrote: post #34

My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question: "Is Paul Newman known for having had problems with alcohol?" All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response: "Paul Newman is not widely known for having had problems with alcohol. While he portrayed charact…

> I use ChatGPT for many tasks every day, but I couldn't fathom that it would get so wrong something so simple. I think we'll have a term like we have for parents/grandparents that believe everything they see on the internet but specifically for people using LLMs.

What is that term?

Re: Recent AI model progress feels mostly like bullshit

#324

Post by bambax »

aezart wrote: post #294
billforsternz wrote: post #238

Earlier quoted context omitted.

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

> I have seen much AI output which is extraordinary it's funny how one serious fail can impact my point of view so dramatically. I feel the same way. It's like discovering for the first time that magicians aren't doing "real" magic, just sleight of hand and psychological tricks. From that point on, it's impossible to be convinced that a future trick is real magic, no matter how impressive it seems. You know it's fake…

I think there is a big divide here. Every adult on earth knows magic is "fake", but some can still be amazed and entertained by it, while others find it utterly boring because it's fake, and the only possible (mildly) interesting thing about it is to try to figure out what the trick is.

I'm in the second camp but find it kind of sad and often envy the people who can stay entertained even though they know better.

Re: Recent AI model progress feels mostly like bullshit

#325

Post by [deleted] »

bglazer wrote: post #156
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Yeah I’m a computational biology researcher. I’m working on a novel machine learning approach to inferring cellular behavior. I’m currently stumped why my algorithm won’t converge. So, I describe the mathematics to ChatGPT-o3-mini-high to try to help reason about what’s going on. It was almost completely useless. Like blog-slop “intro to ML” solutions and ideas. It ignores all the mathematical context, and zeros in o…

[deleted]

Re: Recent AI model progress feels mostly like bullshit

#326

Post by otabdeveloper4 »

bambax wrote: post #322
otabdeveloper4 wrote: post #319

Earlier quoted context omitted.

Anecdotally: schoolkids are at the leading edge of LLM innovation, and nowadays all homework assignments are explicitly made to be LLM-proof. (Well, at least in my son's school. Yours might be different.) This effectively makes LLMs useless for education. (Also sours the next generation on LLMs in general, these things are extremely lame to the proverbial "kids these days".)

How do you make homework assignments LLM-proof? There may be a huge business opportunity if that actually works, because LLMs are destroying education at a rapid pace.

You just (lol) need to give non-standard problems and demand students to provide reasoning and explanations along with the answer. Yeah, LLMs can "reason" too, but it's obvious when the output comes from an LLM here.

(Yes, that's a lot of work for a teacher. Gone are the days when you could just assign reports as homework.)

Re: Recent AI model progress feels mostly like bullshit

#327

Post by famouswaffles »

tsimionescu wrote: post #278
famouswaffles wrote: post #251

Earlier quoted context omitted.

Gradient descent is a dumb optimizer. LLM training is not at all like a human reading a book and more like evolution tuning adaptations over centuries. You would not expect either process to be aware of anything they are converging towards. So having lots of books that talk about chess in training will predictably just return a model that knows how to talk about chess really well. I'm not surprised they may know how…

The whole premise on which the immense valuations of these AI companies is based on is that they are learning general reasoning skills from their training on language. That is, that simply training on text is going to eventually give the AI the ability to generate language that reasons at more or less human level in more or less any domain of knowledge. This whole premise crashes and burns if you need task-specific t…

>The whole premise on which the immense valuations of these AI companies is based on is that they are learning general reasoning skills from their training on language.

And they do, just not always in the ways we expect.

>This whole premise crashes and burns if you need task-specific training, like explicit chess training.

Everyone needs task specific training. Any human good at chess or anything enough to make it a profession needs it. So I have no idea why people would expect any less for a Machine.

>then we have to accept that they can't learn, say, to sell business software unless they include business software pitches in the training set, and there are going to be FAR fewer of those than chess games.

Yeah so ? How much business pitches they need in the training set has no correlation with chess. I don't see any reason to believe what is already present isn't enough. There's enough chess data on the internet to teach them chess too, it's just a matter of how much open AI care about it.

Re: Recent AI model progress feels mostly like bullshit

#328

Post by famouswaffles »

suddenlybananas wrote: post #310
famouswaffles wrote: post #258

Earlier quoted context omitted.

It's implausible that fine-tuning of a premier model would have anywhere near that turn around time. Even if they wanted to and had no qualms doing so, it's not happening anywhere near that fast.

It's really not that implausible, they probably are adding stuff to the data-soup all the time and have a system in place for it.

Yeah it is lol. You don't just train your model on whatever you like when you're expected to serve it. They're are a host of problems with doing that. The idea that they trained on this obscure benchmark released about the day of is actually very silly.

Re: Recent AI model progress feels mostly like bullshit

#329

Post by throwawayffffas »

I agree, about both the issue with benchmarks not being relevant to actual use cases and the "wants to sound smart" issue. I have seen them both first hand interacting with llms.

I think the ability to embed arbitrary knowledge written in arbitrary formats is the most important thing llms have achieved.

In my experience trying to get an llm to perform a task as vast and open ended as the one the author describes is fundamentally misguided. The llms were not trained for that and won't be able to do it in a satisfactory degree. But all this research has thankfully provided us with the software and hardware tools where one could start working on training a model that can.

Contrast that to 5-6 years ago, when all you could hope for this kind of thing was simple rule based and pattern matching systems.

Re: Recent AI model progress feels mostly like bullshit

#330

Post by DebtDeflation »

ants_everywhere wrote: post #65

There are real and obvious improvements in the past few model updates and I'm not sure what the disconnect there is. Maybe it's that I do have PhD level questions to ask them, and they've gotten much better at it. But I suspect that these anecdotes are driven by something else. Perhaps people found a workable prompt strategy by trial and error on an earlier model and it works less well with later models. Or perhaps t…

The issue is the scale of the improvements. GPT-3.5 Instruct was an utterly massive leap over everything that came before it. GPT-4 was a very big jump over that. Everything since has seemed incremental. Yes we got multimodal but that was part of GPT-4, they just didn't release it initially, and up until very recently it mostly handed off to another model. Yes we got reasoning models, but people had been using CoT for awhile so it was just a matter of time before RL got used to train it into models. Witness the continual delays of GPT-5 and the back and forth on whether it will be its own model or just a router model that picks the best existing model to hand a prompt off to.
Post reply on HN