Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

361–370 of 370 posts

Re: Small Models Have Arrived

#361

Post by schrodinger »

jimmaswell wrote: post #91
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

If anyone else finds that BitterLesson link difficult to read because it wraps full-width, you can type "javascript:" in your URL bar and paste:

    document.body.style.cssText='margin:auto;width:600px;'
(Chrome strips the "javascript:" for security if I were to include it myself.)

Re: Small Models Have Arrived

#362

Post by iainmerrick »

Dylan16807 wrote: post #208
iainmerrick wrote: post #191

Earlier quoted context omitted.

I think you have it backwards. The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data. Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than…

> I think you have it backwards. > maybe if we use a blend of raw data and hand-crafted heuristics I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.

I was replying to this:

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games.

If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”

Re: Small Models Have Arrived

#363

Post by Dylan16807 »

iainmerrick wrote: post #362
Dylan16807 wrote: post #208

Earlier quoted context omitted.

> I think you have it backwards. > maybe if we use a blend of raw data and hand-crafted heuristics I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.

I was replying to this: The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”

Hand-crafted even if you're feeding in the raw commentary? That seems like a weird way to consider it. Wouldn't that make LLMs in general "hand-crafted"?

And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.

Re: Small Models Have Arrived

#364

Post by theendisney »

jodrellblank wrote: post #332
59nadir wrote: post #310

Earlier quoted context omitted.

Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious. For analysis, bug h…

> " I have yet to see anything more useful in terms of code generation " Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last lo…

Just like with human langages their only advantage is the community around it, the culture. (Im pretending size is a part of that Obj) No one does it but that is how you should pick it.

Re: Small Models Have Arrived

#365

Post by xnx »

schrodinger wrote: post #361
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

If anyone else finds that BitterLesson link difficult to read because it wraps full-width, you can type "javascript:" in your URL bar and paste: document.body.style.cssText='margin:auto;width:600px;' (Chrome strips the "javascript:" for security if I were to include it myself.)

Or use reader mode or resize the browser window.

Re: Small Models Have Arrived

#366

Post by mdh6 »

apatheticonion wrote: post #197
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable. DeepSeek v4 flash has been dirt cheap and so fast that my development loop is; - small prompt - review - small prompt - review I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs…

[dead]

Re: Small Models Have Arrived

#367

Post by zahlman »

Foobar8568 wrote: post #97
zahlman wrote: post #90

Earlier quoted context omitted.

> when the Opuses are still objectively a bargain for the power they bring. The cost isn't just what you're billed. There are security, privacy etc. concerns.

I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.

If you're writing open-source code then there's obviously nothing wrong with publishing it in a public repo. "Using Github" doesn't require you to use their CI, but even then, a human managing secrets for GitHub CI is worlds apart from trying to make sure an internet-connected agent doesn't leak secrets. And if you have sensitive data that you can't send to a remote model but you would benefit from the technology, then processing it with a local model without network access is the obvious way to address that.

Re: Small Models Have Arrived

#368

Post by EagnaIonat »

giraffe_lady wrote: post #345
EagnaIonat wrote: post #301

Earlier quoted context omitted.

> Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, Try using a LLM model for RAG embeddings and get back to us on that.

Why do you think I haven't done this?

If you have, then it explains so much more.

Re: Small Models Have Arrived

#369

Post by rurban »

rcktmrtn wrote: post #344
blueferret wrote: post #342

Earlier quoted context omitted.

Several exist actually. Try whichllm.app or fitmyllm.com.

> https://www.whichllm.app/ - Linux, general use case, balance - 16 GB RAM - 10 GB VRAM Recommendation: Kimi-K3 This checks out.

whichllm sucks. It only recommends gguf models, and llama.cpp is not supported.

fitmyllm is much better

Re: Small Models Have Arrived

#370

Post by rurban »

imp0cat wrote: post #274
krisoft wrote: post #46

Earlier quoted context omitted.

And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.

That plane was destroyed by Russia, wasn't it? I believe it was partially disassembled when Russia invaded Ukraine and so it wasn't possible to save it. :(

2 Ukrainian Antonovs made it to Germany, Leipzig, and are doing well flying weapons in
Post reply on HN