Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

111–120 of 804 posts

Re: Local AI needs to be the norm

#111

Post by avidphantasm »

js8 wrote: post #89
oytis wrote: post #27

Earlier quoted context omitted.

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

What is the business model of Wikipedia? I don't think there is any. Not everything good in our society needs to have a "business model". People still work on it. It's FINE.

Ultimately, information is a public good: it is non-excludable (you can’t stop people from using it) and it is non-rival (we can all use it at the same time). Public goods are often very useful, and because they are non-excludable and non-rival, ultimately can’t have a market-based business model. I would class open-weights AI models as public goods, and would support government expenditure to produce them.

Re: Local AI needs to be the norm

#114

Post by nullc »

dakolli wrote: post #90
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

> two 4090s is not consumer grade

I think that is a very narrow perspective. Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"?

I agree with your view that cheap tokens on SOTA are a trap-- people should use local AI or no AI.

Re: Local AI needs to be the norm

#115

Post by dakolli »

hparadiz wrote: post #100
dakolli wrote: post #90

Earlier quoted context omitted.

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

Posts like this are so funny to me. I'm staring at a mountain of old hardware right now that cost about $20k ten years ago. I have to pay someone now to come haul it away. What makes you think the current new hardware won't end up with the same fate. > Just write your own fkin code people Bro is nostalgic for googling random stack overflow threads for 10 days to figure out a bug the agent fixes in an hour.

I'm just saying that agent that can fix your bugs actually cost $100-150 an hour to run and you're getting it essentially for $200.00 a month.

The cost of cloud compute actually hasn't gone down for old hardware all that much, it still costs $500.00 a year rent 4 core i7700k that's 10 years old. Don't expect much more valuable hardware, like modern GPUs to deflate in price all that quickly.

There's 3 fabs in the world that make ddr7 and they aren't going to be selling their stock to consumers going forward, it will be purchased by datacenters almost entirely and stay in them until EOL.

Your brain is going to atrophy (this is proven), they'll raise the price to something thats closer to break even and you'll be forced to pay it because you no longer have those muscles.

Re: Local AI needs to be the norm

#116

Post by [deleted] »

zozbot234 wrote: post #102
dakolli wrote: post #90

Earlier quoted context omitted.

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade…

No one runs SOTA models 24/7 for individual use or even for a single household or small business, whereas you can run your own hardware basically 24/7 for AI inference. With the new DeepSeek V4 series and its uniquely memory-light KV cache you can even extend this to parallel inference in order to hide memory bandwidth bottlenecks and increase compute intensity. This is perhaps not so useful on a 128GB or 96GB RAM Ap…

[deleted]

Re: Local AI needs to be the norm

#117

Post by franze »

hackyhacky wrote: post #110

I would like a standardized API for local AI to exist outside of the Apple ecosystem. The Prompt API is Chrome is halfway there. * What is the answer to local AI for native apps on Windows? * What is the answer to local AI for Linux? This is a big opportunity for Linux, given the high quality of open-weight models. I hope some answer emerges before designs fracture and we get a dozen mutually incompatible answers.

i researched that question for apfel https://github.com/Arthur-Ficial/apfel and standardized API is openai api so thats what i went with

Re: Local AI needs to be the norm

#119

Post by avadodin »

aabhay wrote: post #52
mattlondon wrote: post #47

Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference. Damned if they do, damned if they don't.

This is a weird take. If its not opt in or you’re shoe horning it into a browser, then that sucks. Nobody is getting enraged that an app for running local LLMs downloads data to do so.

Although you can opt out and even disable the download feature when you build them in some cases, most of the local LLM tools are too download–happy by default.

Re: Local AI needs to be the norm

#120

Post by phainopepla2 »

js8 wrote: post #89
oytis wrote: post #27

Earlier quoted context omitted.

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

What is the business model of Wikipedia? I don't think there is any. Not everything good in our society needs to have a "business model". People still work on it. It's FINE.

Training AI models is capital intensive, though. Unless there's some sort of mega-crowdfunding effort for open weight model training there needs to be a way to recoup that money on the other end. Either that or state sponsorship I guess
Post reply on HN