Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

221–230 of 371 posts

Re: Small Models Have Arrived

#221

Post by bsder »

wild_egg wrote: post #124
klipt wrote: post #108

Earlier quoted context omitted.

Perhaps an analogy to Moore's law? Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner. What countered it: Moore's law stopped working. Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.

Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped.

You are technically correct. The best kind of correct.

However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.

And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.

Re: Small Models Have Arrived

#222

Post by shepherdjerred »

teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

I choose to use Luna for most tasks because it is cost efficient, even though I get a pretty generous budget from my company.

Sometimes I will use Fable or Sol for large features/projects, or research/exploration.

I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.

Re: Small Models Have Arrived

#223

Post by zmmmmm »

The "good enough" concept is interesting because of how systematically people over estimate it. So often, things that are lower quality but thought to be "good enough" turn out to be either not good enough or not worth it compared to just using the higher quality "thing".

I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.

Re: Small Models Have Arrived

#224

Post by edot »

fragmede wrote: post #211
edot wrote: post #209

Earlier quoted context omitted.

Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.

ChatGPT does not have stockfish as a tool it can call.

Yeah but it can just install it. It writes arbitrary code. It can do whatever you want it to do.

Re: Small Models Have Arrived

#225

Post by camuel »

nickysielicki wrote: post #168
Evidlo wrote: post #137

Earlier quoted context omitted.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

It's the exact opposite.

The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

Re: Small Models Have Arrived

#226

Post by andsoitis »

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?"

What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.

Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.

Re: Small Models Have Arrived

#228

Post by mw888 »

keeda wrote: post #120
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then. However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating som…

It does have to be said that if LLMs keep becoming better coders at some point the bottleneck on quality is prompting. Good ideas have many hidden assumptions you think are procedural but often are pivotal to your broader vision.

I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt.

It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.

Re: Small Models Have Arrived

#229

Post by inigyou »

camuel wrote: post #225
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

It's the exact opposite. The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks. Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, den…

Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.

Re: Small Models Have Arrived

#230

Post by klipt »

camuel wrote: post #225
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

It's the exact opposite. The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks. Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, den…

But the harness still matters.

In the case of stockfish, the harness is a tree search around the neural network evaluations.

Post reply on HN