Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

181–190 of 804 posts

Re: Local AI needs to be the norm

#181

Post by rrvsh »

giobox wrote: post #155
apublicfrog wrote: post #147

Earlier quoted context omitted.

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

> they don't have a used by date For quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever. Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc. h…

Nobody is unaware of the knowledge cutoff, and sharing the Wikipedia article is not helping anyone. Your point is easily rebutted by taking whatever open weights/source model has an outdated cutoff and training or fine tuning it on more data, which is again always going to be viable given a modicum of compute

Re: Local AI needs to be the norm

#182

Post by furyofantares »

TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

What's the gamble here exactly? What agency do we have in it right now?

Re: Local AI needs to be the norm

#183

Post by vb-8448 »

lelanthran wrote: post #168
vb-8448 wrote: post #150

Earlier quoted context omitted.

My experience is that in medium/big codebases even with single functions going with the xhigh is basically better from a user perspective (faster to get the result, and you can trust it) while going with lower models(e.g. sonnet instead of opus) you have to always carefully review the output because 1 of 10 it will hallucinate, you won't catch it immediately and at some point it will bite you.

> My experience is that in medium/big codebases even with single functions going with the xhigh is basically better from a user perspective (faster to get the result, and you can trust it) while going with lower models(e.g. sonnet instead of opus) you have to always carefully review the output because 1 of 10 it will hallucinate, What do you mean "trust it"? It sounds like you want to vibe-code (never look at the out…

With SOTA models I can just set up the instructions (even a little bit fuzzy), go away for 10 or 15 minutes, come back and just check result and adjust when necessary (and most of the time small adjustment are necessary, but the overall work is pretty good).

With subpar models I must be more careful on providing instructions and check it step by step because the path it chose is wrong, or I didn't ask for or the agent stuck in a loop somewhere.

Re: Local AI needs to be the norm

#184

Post by tcp_handshaker »

giobox wrote: post #155
apublicfrog wrote: post #147

Earlier quoted context omitted.

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

> they don't have a used by date For quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever. Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc. h…

You could learn how to code...a whole generation did it before...

Re: Local AI needs to be the norm

#185

Post by mystraline »

2ndorderthought wrote: post #106
wood_spirit wrote: post #65

Earlier quoted context omitted.

Meta released Llama just when OpenAI was so hot and its valuation was going through the roof. Speculating, but Meta probably thought the model not competitive enough to keep as a secret weapon but well good enough to commercially damage OpenAI who were a sudden competitor for most-valued-company? In the same way you can imagine the Chinese government pushing the release of deepseek etc to make sure no one thinks the…

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

Thats because the USA has really nothing big to export. Yay, designs.

China? Im getting ready to watch the URKL (universal robot knockout league) go on. The USA is dicking around with failed robot dogs.

The USA has been a failed country, coasting on massive inertia. But the tech avenues from a article I cant find showed the USA 8/64 areas excelling. China was 56/64 areas excelling.

Re: Local AI needs to be the norm

#186

Post by dleslie »

CamperBob2 wrote: post #179
dleslie wrote: post #91

Earlier quoted context omitted.

This is where government funding can play a role. Sometimes there are things where the public good is best served with public expenditure.

"Government funding" these days would mean that Trump pays Elon Musk (or more likely vice versa) to make Grok 4.20 the only legal LLM for use by Americans.

Outside of the USA it would not look like a wealth transfer to an oligarch.

Not every country is in a crypto-libertarian race to hoard power and wealth.

Re: Local AI needs to be the norm

#187

Post by reissbaker »

CamperBob2 wrote: post #157
dakolli wrote: post #90

Earlier quoted context omitted.

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. Not if you're OK with 4-bit quantization. More like $30K-$50K one time. Spring for 8 RTX6000s instead of 4, and you can use the full-precision K2.6 weights ( https://github.com/local-inference-lab/rtx6kpro/blob/master/... ).

RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works.

I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usable perf fell to only $10k, cloud ones will be way faster and you'd need a lot of tokens to break even.

Re: Local AI needs to be the norm

#189

Post by michaelje »

TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

[dead]

Re: Local AI needs to be the norm

#190

Post by nightski »

apublicfrog wrote: post #147
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

Hardware. Frontier labs are driving up demand so much that it's priced significantly above cost making it far less affordable. Just look at Nvidia's profit margins.
Post reply on HN