Live data from Hacker News

Small Models Have Arrived

calv.info

131–140 of 374 posts

Re: Small Models Have Arrived

#131

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

I'd respectfully push back on the framing here.

If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)

But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain

Not every output needs to be the best to produce value.

And as specialist models increase in cost, their cost/value proposition goes down.

At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.

That's where fine tuning helps.

Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.

Re: Small Models Have Arrived

#132
post #49

Earlier quoted context omitted.

A single, used 3090 costs more than I have ever spent on a computer.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.

Re: Small Models Have Arrived

#133
Watch reasoning tokens though. We tried a small reasoning model that burned ~2800 thinking tokens per call, 3x the cost of a cheaper non-reasoning one despite a better price sheet.

Re: Small Models Have Arrived

#135
post #15
post #12

Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local…

You should be glad to know digital ocean already offers this

In theory so does AWS, but the Bedrock model selection is badly in need of a refresh.

Re: Small Models Have Arrived

#136

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence. VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger. The Bitter Lesson is a valid and trenchant observation about how about we got here, but…

> VibeThinker 3B constitutes extraordinary evidence.. math and logical reasoning

Any similar model aimed at coding?

A >10B model for mass spawning/swarming and reporting back to a larger model

Re: Small Models Have Arrived

#137

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.

I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

Re: Small Models Have Arrived

#138

Earlier quoted context omitted.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.

The current supply chain problems will eventually pass.

Re: Small Models Have Arrived

#139

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence. VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger. The Bitter Lesson is a valid and trenchant observation about how about we got here, but…

So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer?

Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?

Re: Small Models Have Arrived

#140

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

Maybe depends on how you ask it? Directly, or let it write a chess program? I think the latter can yield way better results.
Post reply on HN