Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

191–200 of 371 posts

Re: Small Models Have Arrived

#191

Post by iainmerrick »

nickysielicki wrote: post #168
Evidlo wrote: post #137

Earlier quoted context omitted.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

I think you have it backwards.

The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.

Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.

Re: Small Models Have Arrived

#192

Post by avadodin »

Foobar8568 wrote: post #97
zahlman wrote: post #90

Earlier quoted context omitted.

> when the Opuses are still objectively a bargain for the power they bring. The cost isn't just what you're billed. There are security, privacy etc. concerns.

I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.

If Orang mane bans GitHub they've got their local clones and can whip out a local server and a CI solution.

If Orang mane bans Claude, they've got their local models.

The latter has already happened too so I'd say their risk modeling is spot on.

Re: Small Models Have Arrived

#193

Post by noodletheworld »

teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

Is sol better?

Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.

Going from sol to luna is a downgrade.

It is not a question, it is a fact.

> Is sol actually worth the extra cost?

Is a question only you can answer, because it has no generic answer.

Right now, for me, being able to use sol is worth the cost, but using it all the time is not.

I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.

Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.

Team will quickly shift. People hate losing access to ai.

Re: Small Models Have Arrived

#194

Post by quotescoreai »

sroussey wrote: post #81
jermaustin1 wrote: post #26

Earlier quoted context omitted.

To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments. For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playi…

I have trouble getting simple extraction to work sometimes. I have a block of text describing people and their roles at a company and their ages, and i asked for structured results of an array of these things with the text span that it appears in and all i can say is: nope.

[flagged]

Re: Small Models Have Arrived

#195

Post by avadodin »

bityard wrote: post #148
viscousviolin wrote: post #79

Earlier quoted context omitted.

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

A small model, yes! But not necessarily a good model. With the additional caveat that I don't know whether that specific card is supported by modern drivers. You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't al…

You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions.

An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.

Re: Small Models Have Arrived

#196

Post by anon373839 »

srcreigh wrote: post #130
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

No. The bitter lesson is about capabilities. GP is talking about efficiency. GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal. Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have ge…

> The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions

The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.

Re: Small Models Have Arrived

#197

Post by apatheticonion »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable.

DeepSeek v4 flash has been dirt cheap and so fast that my development loop is;

- small prompt

- review

- small prompt

- review

I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding).

In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine).

I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself.

That said, most of my team vibe codes and reviewing their work is like pulling teeth.

Re: Small Models Have Arrived

#198

Post by kridsdale1 »

NickNaraghi wrote: post #13

> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about…

I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.

Re: Small Models Have Arrived

#199

Post by a13n »

Regarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?

Re: Small Models Have Arrived

#200

Post by [deleted] »

nickysielicki wrote: post #128
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

[deleted]
Post reply on HN