Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

61–70 of 371 posts

Re: Small Models Have Arrived

#61

Post by casper14 »

LoveMistral wrote: post #21
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3

What are some limitations you have found with using a smaller model like that?

Re: Small Models Have Arrived

#63

Post by riazrizvi »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

Re: Small Models Have Arrived

#64

Post by DennisP »

LPisGood wrote: post #16
michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.

Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.

Re: Small Models Have Arrived

#65

Post by 9cb14c1ec0 »

thayne wrote: post #49
jermaustin1 wrote: post #42

Earlier quoted context omitted.

Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that…

A single, used 3090 costs more than I have ever spent on a computer.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

Re: Small Models Have Arrived

#66

Post by breezybottom »

trvz wrote: post #19
hartator wrote: post #18

I have trouble seeing the points of using less capable models. I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.

First, smaller models are fun for hackers: you can run them locally, or run them faster. Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.

If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.

Re: Small Models Have Arrived

#67

Post by sroerick »

dzonga wrote: post #50

small models + a good application layer - are more than enough, good for routine business tasks. the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.

Can you elaborate on this?

Re: Small Models Have Arrived

#68

Post by trvz »

breezybottom wrote: post #66
trvz wrote: post #19

Earlier quoted context omitted.

First, smaller models are fun for hackers: you can run them locally, or run them faster. Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.

If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.

You misunderstood what I was referring to by “hacker” there.

Re: Small Models Have Arrived

#70

Post by dgellow »

riazrizvi wrote: post #63
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
Post reply on HN