Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

21–30 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#21

Post by dghlsakjg »

paulsutter wrote: post #11

Im able to get substantially more coding done than three months ago. This could be largely in the tooling (coding agents, deep research). But the models are better too, for both coding and brainstorming. And tooling counts, to me, as progress. Learning to harness current tools helps to harness future tools. Work on projects that will benefit from advancements, but can succeed without them.

I'm not sure if I'm able to do more of the hard stuff, but a lot of the easy but time consuming stuff is now easily done by LLMs.

Example: I frequently get requests for data from Customer Support that used to require 15 minutes of my time noodling around writing SQL queries. I can cut that down to less than a minute now.

Re: Recent AI model progress feels mostly like bullshit

#22

Post by maccard »

My experience as someone who uses LLMs and a coding assist plugin (sometimes), but is somewhat bearish on AI is that GPT/Claude and friends have gotten worse in the last 12 months or so, and local LLMs have gone from useless to borderline functional but still not really usable for day to day.

Personally, I think the models are “good enough” that we need to start seeing the improvements in tooling and applications that come with them now. I think MCP is a good step in the right direction, but I’m sceptical on the whole thing (and have been since the beginning, despite being a user of the tech).

Re: Recent AI model progress feels mostly like bullshit

#23

Post by mentalgear »

shostack wrote: post #15
fxtentacle wrote: post #3

I'd say most of the recent AI model progress has been on price. A 4-bit quant of QwQ-32B is surprisingly close to Claude 3.5 in coding performance. But it's small enough to run on a consumer GPU, which means deployment price is now down to $0.10 per hour. (from $12+ for models requiring 8x H100)

Yeah, I'm thinking of this from a Wardley map standpoint. What innovation opens up when AI gets sufficiently commoditized?

Brute force, brute force everything at least for the domains you can have automatic verification in.

Re: Recent AI model progress feels mostly like bullshit

#24

Post by xiphias2 »

fxtentacle wrote: post #3

I'd say most of the recent AI model progress has been on price. A 4-bit quant of QwQ-32B is surprisingly close to Claude 3.5 in coding performance. But it's small enough to run on a consumer GPU, which means deployment price is now down to $0.10 per hour. (from $12+ for models requiring 8x H100)

Have you compared it with 8-bit QwQ-17B?

In my evals 8 bit quantized smaller Qwen models were better, but again evaluating is hard.

Re: Recent AI model progress feels mostly like bullshit

#26

Post by bredren »

shostack wrote: post #15
fxtentacle wrote: post #3

I'd say most of the recent AI model progress has been on price. A 4-bit quant of QwQ-32B is surprisingly close to Claude 3.5 in coding performance. But it's small enough to run on a consumer GPU, which means deployment price is now down to $0.10 per hour. (from $12+ for models requiring 8x H100)

Yeah, I'm thinking of this from a Wardley map standpoint. What innovation opens up when AI gets sufficiently commoditized?

One thing I’ve seen is large enterprises extracting money from consumers by putting administrative burden on them.

For example, you can see this in health insurance reimbursements and wireless carriers plan changes. (ie, Verizon’s shift from Do More, etc to what they have now)

Companies basically set up circumstances where consumers lose small amounts of money on a recurring basis or sporadically enough that the people will just pay the money rather than a maze of calls, website navigation and time suck to recover funds due to them or that shouldn’t have been taken in the first place.

I’m hopeful well commoditized AI will give consumers a fighting chance at this and other types of disenfranchisement that seems to be increasingly normalized by companies that have consultants that do nothing but optimize for their own financial position.

Re: Recent AI model progress feels mostly like bullshit

#27

Post by throw310822 »

I hope it's true. Even if LLMs development stopped now, we would still keep finding new uses for them at least for the next ten years. The technology is evolving way faster than we can meaningfully absorb it and I am genuinely frightened by the consequences. So I hope we're hitting some point of diminishing returns, although I don't believe it a bit.

Re: Recent AI model progress feels mostly like bullshit

#28

Post by maccard »

ohgr wrote: post #8

It’s not even approaching the asymptotic line of promises made at any achievable rate for the amount of cash being thrown at it. Where’s the business model? Suck investors dry at the start of a financial collapse? Yeah that’s going to end well…

> where’s the business model?

For who? Nvidia sell GPUs, OpenAI and co sell proprietary models and API access, and the startups resell GPT and Claude with custom prompts. Each one is hoping that the layer above has a breakthrough that makes their current spend viable.

If they do, then you don’t want to be left behind, because _everything_ changes. It probably won’t, but it might.

That’s the business model

Re: Recent AI model progress feels mostly like bullshit

#29

Post by usaar333 »

gundmc wrote: post #5

This was published the day before Gemini 2.5 was released. I'd be interested if they see any difference with that model. Anecdotally, that is the first model that really made me go wow and made a big difference for my productivity.

Ya, I find this hard to imagine aging well. Gemini 2.5 solved (at least much better than) multiple real world systems questions I've had in the past that other models could not. Its visual reasoning also jumped significantly on charts (e.g. planning around train schedules)

Even Sonnet 3.7 was able to do refactoring work on my codebase sonnet 3.6 could not.

Really not seeing the "LLMs not improving" story

Re: Recent AI model progress feels mostly like bullshit

#30

Post by jchw »

joelthelion wrote: post #25

I've used gemini 2.5 this weekend with aider and it was frighteningly good. It probably depends a lot on what you are using them for, and in general, I think it's still too early to say exactly where LLMs will lead us.

I think overall quality with Gemini 2.5 is not much better than Gemini 2 in my experience. Gemini 2 was already really good, but just like Claude 3.7, Gemini 2.5 goes some steps forward and some steps backwards. It sometimes generates some really verbose code even when you tell it to be succinct. I am pretty confident that if you evaluate 2.5 for a bit longer you'll come to the same conclusion eventually.
Post reply on HN