Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

161–170 of 373 posts

Re: Small Models Have Arrived

#161

Post by gozzoo »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

> We have, right now, access to things that 10-20 years ago would be considered magic

These things would be considered magic even 4 years ago!

Re: Small Models Have Arrived

#162

Post by kingstnap »

teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

Smaller models + more effort has strong diminishing returns, especially if your goal is to save money.

Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.

Re: Small Models Have Arrived

#163

Post by zamadatix »

sroussey wrote: post #82
zamadatix wrote: post #72

Earlier quoted context omitted.

I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.

where? i would love that.

"Where'd I buy it" or "where is it now" ;)?

It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.

Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.

Re: Small Models Have Arrived

#164

Post by anuptalwalkar »

I kind of agree with your assessment. Running models not just on local, but cheap and lightweight frameworks will drive the next phase.

Not trying to plug, but I do't know any other way. I wrote a piece couple of days ago on small models and memory usage on the edge devices- https://polign.com/blog-edge-agent-memory and https://news.ycombinator.com/item?id=49450816 closing on the same problem.

Re: Small Models Have Arrived

#165

Post by verdverm »

fitsumbelay wrote: post #159

using small local models - with a little bit of extra work - for the first time over the past few days was _really_ illuminating and inspired similar thoughts about how far you can practically get with so little. column of zap emojis, mane ...

I only use open models now, I really think the era of open models is upon us, big or small, but I also agree small models reached the point where you don't have hand hold them with qwen 3.8 27B

Re: Small Models Have Arrived

#166

Post by verdverm »

mathgenius wrote: post #157
throwaway63467 wrote: post #24

I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the…

Can we stick one of these in something that looks and sounds like HAL 9000 ?

with wireless tech, you can embody Ai in just about anything, waiting for that hacker post about turning their toaster into a thinking machine, I have a Anki vector I've been meaning to do this with (has camera, speakers, microphone, and screen built in)

Re: Small Models Have Arrived

#167

Post by QuercusMax »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.

Re: Small Models Have Arrived

#168

Post by nickysielicki »

Evidlo wrote: post #137
nickysielicki wrote: post #128

Earlier quoted context omitted.

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

Re: Small Models Have Arrived

#170

Post by jbstack »

LPisGood wrote: post #16
michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.

Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.
Post reply on HN