Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

381–390 of 804 posts

Re: Local AI needs to be the norm

#381

Post by apublicfrog »

suika wrote: post #162
apublicfrog wrote: post #147

Earlier quoted context omitted.

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

The use cases in the future will be nothing like the use cases from today.

Maybe. The use cases people primarily use LLMs for (documents, coding, design, research) existed decades ago with different tooling. Who knows if the future will have a slew of new problems that require new models or will continue to be similar?

Re: Local AI needs to be the norm

#382

Post by hparadiz »

platevoltage wrote: post #362
hparadiz wrote: post #152

Earlier quoted context omitted.

The architectural problems I deal with day in day out leave no room for atrophy. This is just cope.

You're going to see major cope once that bargain $200/month plan goes away, and every person or company that has embedded these services into their workflows gets to see their actual costs.

Have you actually tried this stuff or are you just saying stuff you hear on the internet?

Re: Local AI needs to be the norm

#383

Post by thefounder »

oytis wrote: post #27
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

Cloud providers have incentives to release open source models but for some reasons this happens only in China. Amazon, Azure, Google benefit from open source models because people run them on their hardware.

Re: Local AI needs to be the norm

#384

Post by pianopatrick »

winocm wrote: post #247
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

Currently I'm testing something like this just to see what happens. I have an old laptop with 4GB of RAM. I attached a USB drive with Gemma 4 31B model (which is 32.6 GB). Currently the laptop is running llama.cpp and trying to respond to a prompt by streaming the model from disk.

The USB drive light is flickering, showing something is happening. It's been about 8 hours since I entered the prompt and I've gotten about 10 tokens back so far. I'm going to leave it running overnight and see what happens.

Re: Local AI needs to be the norm

#386

Post by janalsncm »

ac29 wrote: post #208
nullc wrote: post #114

Earlier quoted context omitted.

> two 4090s is not consumer grade I think that is a very narrow perspective. Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? I agree with your view that cheap tokens on SOTA are a trap-- people should use local AI or no AI.

> Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? $50k is a median priced car in the US. I'd guess >99.9% of people do not own $4000 of GPUs. I consider myself a computer person and I dont think I even own $4000 of computer hardware in total

Fwiw you can finance a car over something like 7 years now. So a lot of people will be paying like $750 per month, not $50k lump sum.

Re: Local AI needs to be the norm

#387

Post by otabdeveloper4 »

reissbaker wrote: post #187
CamperBob2 wrote: post #157

Earlier quoted context omitted.

It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. Not if you're OK with 4-bit quantization. More like $30K-$50K one time. Spring for 8 RTX6000s instead of 4, and you can use the full-precision K2.6 weights ( https://github.com/local-inference-lab/rtx6kpro/blob/master/... ).

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works. I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usa…

> higher param count models will remain smarter for a looong time

They're not smarter, they just know more stuff.

You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM.

The "smarts" comes from post-training, especially around tool use.

Re: Local AI needs to be the norm

#388

Post by janalsncm »

adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

Reasoning over a large codebase is only one use case for large models. For the use cases in the article (summarizing, classifying, basic text rewrites) most phones can handle them just fine.

Re: Local AI needs to be the norm

#389

Post by janalsncm »

peab wrote: post #367
rmunn wrote: post #366

For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model f…

Civit ai is like 99% porn though. Most production usage of image gen is google or open ai as they are by far the best

Google and OpenAI are good for one-offs but if you want a consistent style you need to use a LoRA.

Re: Local AI needs to be the norm

#390

Post by HWR_14 »

hparadiz wrote: post #100
dakolli wrote: post #90

Earlier quoted context omitted.

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

Posts like this are so funny to me. I'm staring at a mountain of old hardware right now that cost about $20k ten years ago. I have to pay someone now to come haul it away. What makes you think the current new hardware won't end up with the same fate. > Just write your own fkin code people Bro is nostalgic for googling random stack overflow threads for 10 days to figure out a bug the agent fixes in an hour.

Do you have any old laptop ram?
Post reply on HN