Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

211–220 of 372 posts

Re: Small Models Have Arrived

#211

Post by fragmede »

edot wrote: post #209
nickysielicki wrote: post #128

Earlier quoted context omitted.

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.

ChatGPT does not have stockfish as a tool it can call.

Re: Small Models Have Arrived

#212

Post by Dylan16807 »

kmeisthax wrote: post #178
Evidlo wrote: post #137

Earlier quoted context omitted.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The Bitter Lesson says that the only things that scale are search and learning. Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson. In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them…

> A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.

Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.

That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.

Re: Small Models Have Arrived

#213

Post by antihipocrat »

nickysielicki wrote: post #168
Evidlo wrote: post #137

Earlier quoted context omitted.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

Re: Small Models Have Arrived

#214

Post by nl »

fragmede wrote: post #211
edot wrote: post #209

Earlier quoted context omitted.

Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.

ChatGPT does not have stockfish as a tool it can call.

Au contraire!

https://mcpmarket.com/server/stockfish

https://til.simonwillison.net/llms/mcp-in-claude-and-chatgpt...

Re: Small Models Have Arrived

#215

Post by brucehoult »

dmoy wrote: post #149
CamperBob2 wrote: post #123

Earlier quoted context omitted.

I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been. In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it. I would also be VERY slow to claim that general-pu…

I don't know anything about the last 8 years of chess engines, but yea maybe 8-10 years ago AlphaZero shit all over e.g. stockfish.

I guess you missed Leela then.

And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.

https://stockfishchess.org/blog/2023/stockfish-16/

Re: Small Models Have Arrived

#216

Post by DrewADesign »

ZainRiz wrote: post #131
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

I'd respectfully push back on the framing here. If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim) But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain Not every output needs t…

At some point, the idea of cost/benefit analysis in the software business turned into the benefit analysis. The amount of money going into the frontier LLM model game is fantastically ridiculous. Being much better than the free resources doesn’t even touch how much better they will have to be to justify the expense of creating them, let alone continually maintaining these services.

Re: Small Models Have Arrived

#217

Post by kevin_thibedeau »

spockz wrote: post #146
oceanplexian wrote: post #100

Earlier quoted context omitted.

Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers. It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.

Spending that kind of moment on a product that gives you personal happiness for years up to decades and then will still have residual worth, which people save up for ages for, is an entirely different proposition than buying a product that may make you faster professionally, but which in the short time can also be achieved by a few dollars worth of subscriptions to a hosted model for even greater effect.

> people save up for ages for

Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.

Re: Small Models Have Arrived

#218

Post by nl »

nickysielicki wrote: post #168
Evidlo wrote: post #137

Earlier quoted context omitted.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games

Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:

> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]

and

> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]

The actual bitter lesson is this:

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]

Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.

Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000

This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Re: Small Models Have Arrived

#219

Post by ianmarcinkowski »

apatheticonion wrote: post #197
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable. DeepSeek v4 flash has been dirt cheap and so fast that my development loop is; - small prompt - review - small prompt - review I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs…

Basically how I feel, but I use the low-powered API models like Sonnet. I feel more energized and focused on the bigger picture than getting stuck on stupid implementation of micro-problems.

I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging.

I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge.

I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.

Re: Small Models Have Arrived

#220

Post by MrDrMcCoy »

cjkaminski wrote: post #179
Animats wrote: post #143

Earlier quoted context omitted.

Good point. Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.

Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.
Post reply on HN