Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

71–80 of 372 posts

Re: Small Models Have Arrived

#71

Post by cyanydeez »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.

Re: Small Models Have Arrived

#72

Post by zamadatix »

jermaustin1 wrote: post #42
__float wrote: post #36

Earlier quoted context omitted.

"with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?

Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that…

I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.

Re: Small Models Have Arrived

#73

Post by arjie »

hartator wrote: post #18

I have trouble seeing the points of using less capable models. I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.

My experience has been that responsiveness is value. For tasks where you need steering, responsiveness allows for better steering. For tasks which you want unattended, better models are just better.

There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.

Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.

Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.

Re: Small Models Have Arrived

#74

Post by cyanydeez »

michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

[flagged]

Re: Small Models Have Arrived

#75

Post by my-next-account »

yousif_123123 wrote: post #6
glimshe wrote: post #2

> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value. Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so. That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance…

[flagged]

Are ya kidding me

Re: Small Models Have Arrived

#76

Post by layer8 »

9cb14c1ec0 wrote: post #65
thayne wrote: post #49

Earlier quoted context omitted.

A single, used 3090 costs more than I have ever spent on a computer.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.

Re: Small Models Have Arrived

#77

Post by swatcoder »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Yes.

The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.

The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

Re: Small Models Have Arrived

#79

Post by viscousviolin »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?
Post reply on HN