Recent AI model progress feels mostly like bullshit • Hacker News Board

Re: Recent AI model progress feels mostly like bullshit

#391

Post by raxxorraxor » Mon, Apr 07, 2025, 3:08 PM UTC

billforsternz wrote: post #238

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

This reminds me of Google quick answers we had for a time in search. It is quite funny if you live outside the US, because it very often got the units or numbers wrong because of different decimal delimiters.

No wonder Trump isn't afraid to put taxes against Canada. Who could take a 3.8 sqare miles country seriously?

Re: Recent AI model progress feels mostly like bullshit

#392

Post by tshaddox » Mon, Apr 07, 2025, 3:11 PM UTC

bambax wrote: post #324

aezart wrote: post #294

Earlier quoted context omitted.

> I have seen much AI output which is extraordinary it's funny how one serious fail can impact my point of view so dramatically. I feel the same way. It's like discovering for the first time that magicians aren't doing "real" magic, just sleight of hand and psychological tricks. From that point on, it's impossible to be convinced that a future trick is real magic, no matter how impressive it seems. You know it's fake…

I think there is a big divide here. Every adult on earth knows magic is "fake", but some can still be amazed and entertained by it, while others find it utterly boring because it's fake, and the only possible (mildly) interesting thing about it is to try to figure out what the trick is. I'm in the second camp but find it kind of sad and often envy the people who can stay entertained even though they know better.

I think magic is extremely interesting (particularly close-up magic), but I also hate the mindset (which seems to be common though not ubiquitous) that stigmatizes any curiosity in how the trick works.

In my view, the trick as it is intended to appear to the audience and the explanation of how the trick is performed are equal and inseparable aspects of my interest as a viewer. Either one without the other is less interesting than the pair.

Re: Recent AI model progress feels mostly like bullshit

#393

Post by boesboes » Mon, Apr 07, 2025, 3:27 PM UTC

iambateman wrote: post #39

The core point in this article is that the LLM wants to report _something_, and so it tends to exaggerate. It’s not very good at saying “no” or not as good as a programmer would hope. When you ask it a question, it tends to say yes. So while the LLM arms race is incrementally increasing benchmark scores, those improvements are illusory. The real challenge is that the LLM’s fundamentally want to seem agreeable, and th…

This rings true. What I notice is that the longer i let Claude work on some code for instance, the more bullshit it invents. I usually can delete about 50-60% of the code & tests it came up with.

And when you ask it to 'just write a test' 50/50 it will try to run it, fail on some trivial issues, delete 90% of your test code and start to loop deeper and deeper into the rabit hole of it's own halliciations.

Or maybe I just suck at prompting hehe

Re: Recent AI model progress feels mostly like bullshit

#394

Post by bglazer » Mon, Apr 07, 2025, 3:38 PM UTC

MoonGhost wrote: post #304

bglazer wrote: post #156

Earlier quoted context omitted.

Yeah I’m a computational biology researcher. I’m working on a novel machine learning approach to inferring cellular behavior. I’m currently stumped why my algorithm won’t converge. So, I describe the mathematics to ChatGPT-o3-mini-high to try to help reason about what’s going on. It was almost completely useless. Like blog-slop “intro to ML” solutions and ideas. It ignores all the mathematical context, and zeros in o…

I was working some time ago on image processing model using GAN architecture. One model produces output and tries to fool the second. Both are trained together. Simple, but requires a lot extra efforts to make it work. Unstable and falls apart (blows up to unrecoverable state). I found some ways to make it work by adding new loss functions, changing params, changing models' architectures and sizes. Adjusting some coe…

Yes, I suspect that engineering the loss and hyperparams could eventually get this to work. However, I was hoping the model would help me get to a more fundamental insight into why the training falls into bad minima. Like the Wasserstein GAN is a principled change to the GAN that improves stability, not just fiddling around with Adam’s beta parameter.

The reason I expected better mathematical reasoning is because the companies making them are very loudly proclaiming that these models are capable of high level mathematical reasoning.

And yes the fact I don’t have to look at matplotlib documentation anymore makes these models extremely useful already, but thats qualitatively different from having Putnam prize winning reasoning ability

Re: Recent AI model progress feels mostly like bullshit

#395

Post by consumer451 » Mon, Apr 07, 2025, 3:42 PM UTC

wg0 wrote: post #312

Unlike many - I find author's complaints on the dot. Once all the AI batch startups have sold subscriptions to the cohort and there's no more further market growth because businesses outside don't want to roll the dice on a probabilistic model that doesn't have an understanding of pretty much anything rather is a clever imitation machine on the content it has seen, the AI bubble will burst when more statups would sta…

I would go even further than TFA. In my personal experience using Windsurf daily, Sonnet 3.5 is still my preferred model. 3.7 makes many more changes that I did not ask for, often breaking things. This is an issue with many models, but it got worse with 3.7.

Re: Recent AI model progress feels mostly like bullshit

#396

Post by aoeusnth1 » Mon, Apr 07, 2025, 3:50 PM UTC

hyperbovine wrote: post #385

aoeusnth1 wrote: post #383

Earlier quoted context omitted.

2.5 pro scores 25%. It’s just a much harder math benchmark which will fall by the end of next year just like all the others. You won’t be vindicated.

Bold claim! Let's see what that 25% is. I guarantee it is the portion of the exam which is trivially answerable if you have a stored database of all previous math exams ever written to consult.

There is 0% of the exam which is trivially answerable.

The entire point of USAMO problems is that they demand novel insight and rigorous, original proofs. They are intentionally designed not to be variations of things you can just look up. You have to reason your way through, step by logical step.

Getting 25% (~11 points) is exceptionally difficult. That often means fully solving one problem and maybe getting solid partial credit on another. The median score is often in the single digits.

Re: Recent AI model progress feels mostly like bullshit

#397

Post by behnamoh » Mon, Apr 07, 2025, 3:52 PM UTC

consumer451 wrote: post #395

wg0 wrote: post #312

Unlike many - I find author's complaints on the dot. Once all the AI batch startups have sold subscriptions to the cohort and there's no more further market growth because businesses outside don't want to roll the dice on a probabilistic model that doesn't have an understanding of pretty much anything rather is a clever imitation machine on the content it has seen, the AI bubble will burst when more statups would sta…

I would go even further than TFA. In my personal experience using Windsurf daily, Sonnet 3.5 is still my preferred model. 3.7 makes many more changes that I did not ask for, often breaking things. This is an issue with many models, but it got worse with 3.7.

3.7 is like a wild horse. you really must ground it with clear instructions. it sucks that it doesn't automatically know that but it's tameable.

Re: Recent AI model progress feels mostly like bullshit

#398

Post by cootsnuck » Mon, Apr 07, 2025, 4:00 PM UTC

consumer451 wrote: post #395

wg0 wrote: post #312

Unlike many - I find author's complaints on the dot. Once all the AI batch startups have sold subscriptions to the cohort and there's no more further market growth because businesses outside don't want to roll the dice on a probabilistic model that doesn't have an understanding of pretty much anything rather is a clever imitation machine on the content it has seen, the AI bubble will burst when more statups would sta…

I would go even further than TFA. In my personal experience using Windsurf daily, Sonnet 3.5 is still my preferred model. 3.7 makes many more changes that I did not ask for, often breaking things. This is an issue with many models, but it got worse with 3.7.

Yea, I've experienced this too with 3.7. Not always though. It has been helpful for me more often than not helpful. But yea 3.5 "felt" better to me.

Part of me thinks this is because I expected less of 3.5 and therefore interacted with it differently.

It's funny because it's unlikely that everyone interacts with these models in the same way. And that's pretty much guaranteed to give different results.

Would be interesting to see some methods come out for individuals to measure their own personal success rate/ productivity / whatever with these different models. And then have a way for people to compare them with each other so we can figure out who is working well with these models and who isn't and figure out why the difference.

Re: Recent AI model progress feels mostly like bullshit

#399

Post by bluefirebrand » Mon, Apr 07, 2025, 4:02 PM UTC

jodrellblank wrote: post #358

bluefirebrand wrote: post #220

Earlier quoted context omitted.

> when there were over a billion people on Earth in 1800 who could have come up with it My point is that humans did come up with it. Humans did not parrot it from someone or something else that showed it to us. We didn't "parrot" splitting the atom. We didn't learn how to build skyscrapers from looking at termite hills and we didn't learn to build rockets that can send a person to the moon from seeing a volcano You a…

It's obvious that humans imitate concepts and don't come up with things de-novo from a blank slate of pure intelligence. So your claim hinges on LLMs parrotting the words they are trained on. But they don't do that, their training makes them abstract over concepts and remix them in new ways to output sentences they weren't trained on, e.g.: Prompt: "Can you give me a URL with some novel components, please?" DuckDuckG…

> fireworks, cannons, jellyfish squeezing water out to accelerate, no sudies of orbits from moons and planets, no chemistry experiments, no inspiration from thousands of years of flamethrowers

Fireworks, cannons, chemistry experiments and flamethrowers are all human inventions

And yes, exactly! We studied orbits of moons and planets. We studied animals like Jellyfish. We choose to observe the world, we extracted data, we experimented, we saw what worked, refined, improved, and succeeded

LLMs are not capable of observing anything. They can only regurgitate and remix the information they are fed by humans! By us, because we can observe

An LLM trained on 100% wrong information will always return wrong information for anything you ask it.

Say you train an LLM with the knowledge that fire can burn underwater. It "thinks" that the step by step instructions for building a fire is to pile wood and then pour water on the wood. It has no conflicting information in its model. It cannot go try to build a fire this way and observe that it is wrong. It is a parrot. It repeats the information that you give it. At best it can find some relationships between data points that humans haven't realized might be related

A human could easily go attempt this, realize it doesn't work, and learn from the experience. Humans are not simply parrots. We are capable of exploring our surroundings and internalizing things without needing someone else to tell us how everything works

> That is, nobody taught humans to split the atom and then humans literally parotted the mechanism and did it, but you attempting to present splitting the atom as a thing which appeared out of nowhere and not remixing any existing concepts is, in your terms, absolute drivel

Building on the work of other humans is not parroting

You outlined the absolute genius of humanity building from first principles all the way to splitting the atom and you still think we're just parroting,

I think we disagree what parroting is entirely.

Re: Recent AI model progress feels mostly like bullshit

#400

Post by jerf » Mon, Apr 07, 2025, 4:06 PM UTC

immibis wrote: post #382

jerf wrote: post #373

Earlier quoted context omitted.

The models have moved on past this working reliably, but an example that I found in the early days of LLMs is asking it "Which is heavier, two pounds of iron or a pound of feathers?" You could very easily trick it into giving the answer about how they're both the same, because of the number of training instances of the well-known question about a pound of each that it encountered. You can still do this to the current…

"Draw a wine glass filled to the brim with wine" worked recently on image generators. They only knew about half-full wine glasses. If you asked a multimodal system questions about the image it just generated, it would tell you the wine was almost overflowing out of the top of the glass. But any trick prompt like this is going to start giving expected results once it gets well-known enough. Late edit: Another one was…

"But any trick prompt like this is going to start giving expected results once it gets well-known enough."

Which makes it difficult to fairly evaluate whether the models have actually gotten better at the feather/iron problem or if it just got enough samples of trick questions that it learned better, either naturally from the internet, or fed as part of the training data. I am fairly certain the training data has had "trick questions" like this added to it, because, I mean, why wouldn't it?

I have noticed in my playing with image AIs that they do seem more prone to getting dragged into local maxima when a human would know the prompt than the LLMs. Perhaps it's all the additional data in an image that reveals it.