Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

351–360 of 373 posts

Re: Small Models Have Arrived

#351

Post by tuesdaynight »

throwaway63467 wrote: post #24

I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the…

Same. I was in the doom AI crowd ~2 years ago, then I realized how many problems could be mitigated or solved by having these local models, and I became part of the cautiously excited crowd

Re: Small Models Have Arrived

#352

Post by sandos »

rcktmrtn wrote: post #344
blueferret wrote: post #342

Earlier quoted context omitted.

Several exist actually. Try whichllm.app or fitmyllm.com.

> https://www.whichllm.app/ - Linux, general use case, balance - 16 GB RAM - 10 GB VRAM Recommendation: Kimi-K3 This checks out.

It says Kimi-K3 for very set of parameters I put in!

64 or 128GB RAM, 6 or 64GB of VRAM....

Not sus at all.

Re: Small Models Have Arrived

#353

Post by samspot »

swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

I received access to AI in May of this year and used smaller models a lot in May, June, and July. Haiku 4.5 is terrible for coding. This month Luna was added and it's incredibly capable. I have to reach for a better model 1/50 times instead of 1/2 or 1/3 with Haiku. Luna is better and cheaper than the Haiku and Sonnet models I've been spending most of my time with.

It's safe to say I didn't come to this conclusion because I was "addicted to chasing frontier models"

Re: Small Models Have Arrived

#354

Post by tencentshill »

sandos wrote: post #352
rcktmrtn wrote: post #344

Earlier quoted context omitted.

> https://www.whichllm.app/ - Linux, general use case, balance - 16 GB RAM - 10 GB VRAM Recommendation: Kimi-K3 This checks out.

It says Kimi-K3 for very set of parameters I put in! 64 or 128GB RAM, 6 or 64GB of VRAM.... Not sus at all.

Roleplay, windows, 16gb/4gb

Kimi-K3

I think the slop site is broken or compromised...

Re: Small Models Have Arrived

#356

Post by bhelkey »

Myrmornis wrote: post #338

Is there a resource somewhere that tells me, "Given that you have X Gb RAM on your laptop, these are the current local models you should consider trying and here's how to configure them to leave enough memory for other applications on your machine."?

Are you familiar with Ollama [1]? It is a particularly easy to use tool to download and run local models. They sort models recent popularity and specify size for the various quantization levels.

I would try using ~1/2 your available ram and iterate from there.

If you have 32GB of RAM, I would give Qwen 3.8 a try. All you would have to do is run "ollama pull qwen3.8:27b" then "ollama run qwen3.8:27b". If you have 16GB of RAM, I would try Gemma 4.

[1] ollama.com

Re: Small Models Have Arrived

#357

Post by bhelkey »

chrishare wrote: post #343
Myrmornis wrote: post #338

Is there a resource somewhere that tells me, "Given that you have X Gb RAM on your laptop, these are the current local models you should consider trying and here's how to configure them to leave enough memory for other applications on your machine."?

Not that I know of, but https://www.canirun.ai/ might be of use

This calls the top coding model for the Apple M1 Pro: qwen2.5-coder-7b, a model released September 18, 2024 [1].

I question this choice. Coding models have improved significantly in the past 2 years.

[1] https://qwen.ai/blog?id=qwen2.5-coder

Re: Small Models Have Arrived

#358

Post by 59nadir »

jodrellblank wrote: post #332
59nadir wrote: post #310

Earlier quoted context omitted.

Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious. For analysis, bug h…

> " I have yet to see anything more useful in terms of code generation " Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last lo…

I think this sounds fantastic, and I welcome all developments in these types of areas where you lean heavily on the hyper-attentive, very fast assistant aspect of LLMs intended to make information more easily searchable, visible and the results more accurate.

I agree that the mismatch between the expectations of OOP as a paradigm and this situation right now feel very similar. I suppose the many billions pumped into selling vibecoding and large-scale code generation via LLMs of other kinds is sort of like the massive amount of money pumped into marketing Java as a viable alternative long before it actually was.

Re: Small Models Have Arrived

#359

Post by crossroadsguy »

blueferret wrote: post #342
Myrmornis wrote: post #338

Is there a resource somewhere that tells me, "Given that you have X Gb RAM on your laptop, these are the current local models you should consider trying and here's how to configure them to leave enough memory for other applications on your machine."?

Several exist actually. Try whichllm.app or fitmyllm.com.

It gives astronomically optimistic results for my 16GB M1 Pro :)

By the way own exploration sort of led me to qwen3.5:9b for the best case scenario balanced model considering almost 10-11GB of RAM is almost always gone anyway. Even with aggressive app quitting.

Post reply on HN