Live data from Hacker News

Local AI needs to be the norm

unix.foo

361–370 of 804 posts

Re: Local AI needs to be the norm

#361
post #311

Earlier quoted context omitted.

Jokes on you. We are already running Deepseekv4Flash, Mimo2.5, MiniMax2.7, Qwen3-397B locally in very affordable hardware. These models are in the real of Opus4.6. For those of us a bit crazy, we are running KimiK2.6, GLM5.1 and more ...

They all still fall short of Opus 4.6, definitely though. They are good but fail on extremely complex tasks, in contrast with a frontier model that will keep on trying until it succeeds or exhausts the solutions space.

Not by much, and moving goalposts makes for a bad comparison. Local open weight models are already more powerful than frontier models from only a year back.

If you believe what you read here, the gap is closing fast.

Re: Local AI needs to be the norm

#362

Earlier quoted context omitted.

I'm just saying that agent that can fix your bugs actually cost $100-150 an hour to run and you're getting it essentially for $200.00 a month. The cost of cloud compute actually hasn't gone down for old hardware all that much, it still costs $500.00 a year rent 4 core i7700k that's 10 years old. Don't expect much more valuable hardware, like modern GPUs to deflate in price all that quickly. There's 3 fabs in the worl…

The architectural problems I deal with day in day out leave no room for atrophy. This is just cope.

You're going to see major cope once that bargain $200/month plan goes away, and every person or company that has embedded these services into their workflows gets to see their actual costs.

Re: Local AI needs to be the norm

#364

Earlier quoted context omitted.

Is it so different? If the US’s fascist experiment continues past the current president, we’ll absolutely be nationalizing frontier companies or exerting equivalent control.

[flagged]

Certainly Biden and Obama check off a few of the 14 points of Fascism, but are we really being serious here? "TDS" is just a thought terminating cliche.

Re: Local AI needs to be the norm

#365

Earlier quoted context omitted.

Anthropic and OpenAI make money on API calls, margins have been reported in public filings. Subs are subsidized.

That's not possible, read my comment above. These are private companies, there are no public filings regarding their profitability in any sense. You're just making things up. If you have a machine running at 150 tok/ps you can only make $5820 a month at $15 per 1mm running 24/7. It costs a hell of a lot more than 6k a month to run Claude 4.7 @ 150 tok/ps on that machine 24/7. This math is a bit off, because you have…

You're forgetting a critical factor: concurrency. If a given hardware serves a single request at 150 tokens/s, it can also serve 20-30 requests at 100 tokens/s. Suddenly your $5K becomes $100K/month, enough to recoup the cost of the hardware in a year or so.

The reason it works: each time you read the model (memory bound) to calculate the next token, you can also update multiple requests (compute bound) while at it. It's also much more energy-efficient per token.

[1] https://aimultiple.com/gpu-benchmark

Re: Local AI needs to be the norm

#366
For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model for it at Civitai, and people doing some very creative things with them. (And also a lot of the obvious things, but it's the Internet, what did you expect?)

I haven't seen a text-based model sharing site spring up yet (perhaps they already have and I don't know about it yet). Civitai, being focused on image-generation, has the obvious advantage that it's easy to show off impressive results from the model on the front page of the website, and judging what someone's home-grown fine-tuned LLM will produce is a lot harder. But at some point I expect a Civitai equivalent site for text models, especially code-based ones, to become popular. That will seriously undercut Anthropic, OpenAI, et al, and will probably force them to find a price equilibrium.

Because once you're competing with "I spend $2,500 up front on a powerful video card, download an open-source model for free, and then I get pretty much everything I need for free" (additional power cost of running that video card isn't nothing, but probably not noticeable in your power bill compared to what you're already using)... then suddenly $200/month means your customers are thinking "after one year I would have been better off with the homegrown solution". The only way they'll continue to pay $200/month is if Claude/GPT/Gemini/whoever is truly head-and-shoulders above the "pay upfront once for hardware then use it for free afterwards" models available. And that's going to be doable, perhaps, but tough.

Re: Local AI needs to be the norm

#367
post #366

For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model f…

Civit ai is like 99% porn though. Most production usage of image gen is google or open ai as they are by far the best

Re: Local AI needs to be the norm

#368
post #367
post #366

For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model f…

Civit ai is like 99% porn though. Most production usage of image gen is google or open ai as they are by far the best

As I said, a lot of the obvious things. And if you're scrolling through the front page without being logged in (i.e., so the default "no mature content" filter is on), there's some really creative stuff being done. I personally like the looks of the RPGv5 (or is the guy up to v6 by now? I forget) model, and plan to use it eventually to create custom portraits of characters in my tabletop roleplay campaigns. (Not running any right now, due to having basically zero free time at the moment, but eventually my current situation will change and I'll have the occasional weekend open again).

But for a site sharing code-generation models, it's a very different scenario. I'm curious to see what will happen in that space.

Re: Local AI needs to be the norm

#369

Earlier quoted context omitted.

In my experience once you get to ~30 gigs of ram for a model like Gemma4, the rest of the 128g of memory is simply nice to have. The speed and costs are what make it tough though, because its slower and more expensive than the same model served on a big accelerator card, and is going to be worse than a frontier model.

I wonder if it really needs to be worse. I am playing with the idea of fine tuning a model on my exact stack and coding patterns. I suspect I could get better performance by training “taste” into a model rather than breadth.

I also wonder about JS only, Python only, etc models.

Maybe the future is a selection of local, specific stack trained models?

Re: Local AI needs to be the norm

#370
This is just emotional rhetoric. Pretty much any app in the last 20 years has depended on a server somewhere, or a cloud provider. Like an AI provider, they can go down, they can turn off if you don't pay your bill, etc.

And local inference requires fairly beefy hardware, that is FAR from ubiquitous across today's userbases. Local models are also still far dumber than what frontier labs can serve.

Weird that this is getting such a tidal wave of upvotes.

Post reply on HN