Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

21–30 of 370 posts

Re: Small Models Have Arrived

#21

Post by LoveMistral »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Same. Mistral 7b has been more than I ever needed for text for years now.

Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.

Mistral 7b can do anything, and it’s basically instant even on an M3

Re: Small Models Have Arrived

#22

Post by shafyy »

hartator wrote: post #18

I have trouble seeing the points of using less capable models. I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.

Some reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment

It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.

Re: Small Models Have Arrived

#23

Post by jmtulloss »

I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.

Re: Small Models Have Arrived

#24

Post by throwaway63467 »

I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation 24/7 (then again I might be wrong). Exciting times.

Re: Small Models Have Arrived

#25

Post by frigidwalnut »

LoveMistral wrote: post #21
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3

Sounds interesting. Can you give more details on your workflow and what tasks you use it for?

Re: Small Models Have Arrived

#26

Post by jermaustin1 »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.

For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.

But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.

Re: Small Models Have Arrived

#27

Post by possibilistic »

> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.

You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?

Re: Small Models Have Arrived

#28

Post by scoring1774 »

jbjbjbjb wrote: post #9
swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is chea…

I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.

Re: Small Models Have Arrived

#29

Post by [deleted] »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

[deleted]
Post reply on HN