Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

221–230 of 804 posts

Re: Local AI needs to be the norm

#221

Post by nullc »

ac29 wrote: post #208
nullc wrote: post #114

Earlier quoted context omitted.

> two 4090s is not consumer grade I think that is a very narrow perspective. Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? I agree with your view that cheap tokens on SOTA are a trap-- people should use local AI or no AI.

> Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is "not consumer"? $50k is a median priced car in the US. I'd guess >99.9% of people do not own $4000 of GPUs. I consider myself a computer person and I dont think I even own $4000 of computer hardware in total

I guess I wasn't clear-- I wasn't so much making the point people do own $4000 in GPUs (though I suspect you are massively underestimating the number who do, also before the current market conditions this would have been more like $2500 in gpus...), but they certainly could per the evidence of car ownership.

A car is super useful, so is an AI. But even if we decide cars are incomparably more useful a great many people pay much more than $4000 over the minimum viable car, and that's money that could be deployed to secure access to private, secure, and autonomous AI facilities. A few thousand dollars in computing is consumer hardware, or at least could easily be with more reason and awareness driving adoption.

People spend a LOT of money in things less useful than local copy of qwen3.6-27b can be.

Re: Local AI needs to be the norm

#222

Post by irishcoffee »

AlexCoventry wrote: post #207
irishcoffee wrote: post #154

Earlier quoted context omitted.

I own 2 5070TI cards in a rig I would gladly donate time to for a distributed training model effort. The kicker is the training data. I would want to gate the data to anything before 2022. I don’t know how to coordinate that, but I would really like to be involved in something like this. SETI, for LLMs.

Bandwidth is the killer, in distributed LLM training.

What’s the rush?

Re: Local AI needs to be the norm

#223

Post by nullc »

ai_fry_ur_brain wrote: post #216
nullc wrote: post #175

Earlier quoted context omitted.

Here is an example-- I'm running hermes + qwen3.6-27b on a workstation GPU (an older RTX A6000 which gets 55tok/s, though people run this model on more limited hardware). A friend an I had previously worked on an entropy extraction scheme and he recently got around to making a writeup about our work: https://wuille.net/posts/binomial-randomness-extractors/ I instructed the agent to read the URL, implement the techniq…

This is maybe the first time Ive seen someone claim to do something useful with such a small model. Congrats, but you're in the 0.0001% thats not just frying their brains, fapping to their local models or doing various magic tricks like a toddler entertained by playing with velcro. At the end of the day you lost an opportunity to improve yourself and excercise your brain, maybe the opportunity cost is worth it idk, b…

This is a change that's been happening gradually over time-- I don't think I could have done this on a local model that could run on a consumer class gpu a couple months ago.

There are plenty of other uses that people have been making for a long time-- e.g. I know someone who uses a fine tuned local model to sort their incoming email and scan their outgoing messages for accidental privacy leaks.

I don't agree with your assessment on an opportunity lost-- I got my reps in on the original work, the AI gave an incremental step forward which made the whole exercise somewhat more valuable to me with minimal additional cost. I think this improves the cost vs benefit in a way that makes me more likely to try other pointless activities, knowing that when I run out of gas I can toss it to AI to try some variations.

Sometimes you're also 27 steps deep on a nested subproblem and you're really just trying to solve sometime. Even in finr craftsmanship not every step needs to be about maximum craftsmanship. :) Sometimes it's just good to get something done.

I think this is much like any other tool. One can carve furniture using only hand tools, but the benefits of a router are hard to dispute. Both approaches exist in the world and sometimes both are used in concert.

As far as people frying their brains with AI -- you don't need local models for that, plenty of people are driving themselves into deep personally and socially destructive delusion just using the chat interfaces.

Re: Local AI needs to be the norm

#224

Post by sillysaurusx »

mystraline wrote: post #185
2ndorderthought wrote: post #106

Earlier quoted context omitted.

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

Thats because the USA has really nothing big to export. Yay, designs. China? Im getting ready to watch the URKL (universal robot knockout league) go on. The USA is dicking around with failed robot dogs. The USA has been a failed country, coasting on massive inertia. But the tech avenues from a article I cant find showed the USA 8/64 areas excelling. China was 56/64 areas excelling.

If this is true, then why are most of the companies that change the world founded in the US?

Re: Local AI needs to be the norm

#225

Post by HDBaseT »

2ndorderthought wrote: post #106
wood_spirit wrote: post #65

Earlier quoted context omitted.

Meta released Llama just when OpenAI was so hot and its valuation was going through the roof. Speculating, but Meta probably thought the model not competitive enough to keep as a secret weapon but well good enough to commercially damage OpenAI who were a sudden competitor for most-valued-company? In the same way you can imagine the Chinese government pushing the release of deepseek etc to make sure no one thinks the…

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

You can still make money on open weight models.

The compute required to run these models is still very far out of reach for the average consumer, yet known enthusiast, therefore they still sell inference, whilst also getting consumer goodwill for providing open weights.

Re: Local AI needs to be the norm

#226

Post by kranke155 »

try-working wrote: post #199
oytis wrote: post #27

Earlier quoted context omitted.

What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more

Open sourcing models is a marketing strategy. Chinese labs and small international labs have no awareness or distribution, so unless they become a hot topic for a while, nobody is going to bother trying out their models. Open source gets them that, and is essentially a tax on newcomers. When you start out you simply have no other option but to open source your models. So, the business model of open models is the same…

China’s long term goal might just be to own the chip layer alongside everything else, and outproduce the US in data centers.

Frontier US labs could still have an advantage for a long time, but many use cases would start gravitating towards Chinese models if they 10x the data centers and provide similar quality inference for a third of the cost.

Re: Local AI needs to be the norm

#227

Post by ai_fry_ur_brain »

nullc wrote: post #223
ai_fry_ur_brain wrote: post #216

Earlier quoted context omitted.

This is maybe the first time Ive seen someone claim to do something useful with such a small model. Congrats, but you're in the 0.0001% thats not just frying their brains, fapping to their local models or doing various magic tricks like a toddler entertained by playing with velcro. At the end of the day you lost an opportunity to improve yourself and excercise your brain, maybe the opportunity cost is worth it idk, b…

This is a change that's been happening gradually over time-- I don't think I could have done this on a local model that could run on a consumer class gpu a couple months ago. There are plenty of other uses that people have been making for a long time-- e.g. I know someone who uses a fine tuned local model to sort their incoming email and scan their outgoing messages for accidental privacy leaks. I don't agree with yo…

I do think post training smaller open source models for very narrow tasks is largely overlooked and there'll be lots of value there if one puts in the effort. However, in a lot of cases we're just compeleting a circle back to deterministic behavior at 1000x the memory/compute requirements just to avoid writing regex.

I agree with you, there's a way to use them responsibly like your router anology, I just think most aren't doing this correctly and its a slippery slope. I'll contend that you probably have used them responsibly in your example.

Re: Local AI needs to be the norm

#228

Post by heydryft »

jmyeet wrote: post #171

I've been looking into options for this and we are getting close. There are two main constraints: memory and memory bandwidth. NVidia segments the market by limiting the amount of memory on GPUs. It currently tops out at 32GB (on a 5090) but it has excellent memory bandwidth (~1.8TB/s). If you want more than the you need to buy an RTX Pro (eg RTX 6000 Pro w/ 96GB for ~$10K) or you get into high high end solutions lik…

[flagged]

Re: Local AI needs to be the norm

#229

Post by ai_fry_ur_brain »

apublicfrog wrote: post #147
TheJCDenton wrote: post #20

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai f…

> It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control. What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They're good enough for 95% of use cases, and they don't have a used by date. From what I can see, the "danger" is not having the next ti…

95% of usecases. What are you smoking.

Re: Local AI needs to be the norm

#230

Post by mrleinad »

mystraline wrote: post #185
2ndorderthought wrote: post #106

Earlier quoted context omitted.

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

Thats because the USA has really nothing big to export. Yay, designs. China? Im getting ready to watch the URKL (universal robot knockout league) go on. The USA is dicking around with failed robot dogs. The USA has been a failed country, coasting on massive inertia. But the tech avenues from a article I cant find showed the USA 8/64 areas excelling. China was 56/64 areas excelling.

China is going to be the next Germany: a loser in the new world without globalization
Post reply on HN