Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

471–480 of 804 posts

Re: Local AI needs to be the norm

#471

Post by watwut »

ThunderSizzle wrote: post #283
Spooky23 wrote: post #245

Earlier quoted context omitted.

Is it so different? If the US’s fascist experiment continues past the current president, we’ll absolutely be nationalizing frontier companies or exerting equivalent control.

[flagged]

> Sigh. Obama and Biden were as every bit "fascist" as Trump.

Absolutely not. There is huge difference in the their behaviors.

> But it's all Trump's fault is much more convenient.

It is not just Trumps fault. Trump is logical consequence of what conservative party became. J.D.Vance and Miller are as much fascists if not more. The whole party worked for this for years and created this.

> And Congress has proven over the last several decades that their oversight is rather meaningless for the goals of American voters rather than special interests.

Of course congress in general is not the place to stop republican party from their fascists goals, because republicans in the congress support Trump 100%. They stand by project 2025 100%. They are doing oversight all right when it comes to blocking democrats.

The idea that the party that made Trump big, promoted ideas he build on and created project 2025 is supposed to be counterbalance to itself is absurd.

Re: Local AI needs to be the norm

#472

Post by watwut »

mystraline wrote: post #185
2ndorderthought wrote: post #106

Earlier quoted context omitted.

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

Thats because the USA has really nothing big to export. Yay, designs. China? Im getting ready to watch the URKL (universal robot knockout league) go on. The USA is dicking around with failed robot dogs. The USA has been a failed country, coasting on massive inertia. But the tech avenues from a article I cant find showed the USA 8/64 areas excelling. China was 56/64 areas excelling.

> Thats because the USA has really nothing big to export. Yay, designs.

USA exports and exported services, especially in IT. And a lot. USA has nothing to export is true only if you intentionally ignore stuff USA exports.

Re: Local AI needs to be the norm

#473

Post by thot_experiment »

amelius wrote: post #458
thot_experiment wrote: post #451

Earlier quoted context omitted.

False. The absolute capability is irrelevant, with the proper harness 31b is more than adequate for a very large portion of the tasks I ask AI to do. The metric isn't how good the model is at Erdos Problems, it's how reliably it can remove drudgery in my life. It just autonomously reverse engineered a bluetooth protocol with minimal intervention, it's ability to react to data and ground itself is constantly impressiv…

This is like saying that 640kB is enough for anybody.

No, it isn't. I am saying that the set of tasks that can be completed by Opus 4.7 has a surprisingly large overlap with the set of tasks that can be completed by Gemma 31B. It is meaningfully equivalent in many cases.

(of course if i'm being honest 640kB is fine, i'm sure tons of the world's commerce is handled by less for example, the delta between a system with 640kb of ram and a modern one is near nil for many people, the UX on a PoS terminal does not require more than that for example, the hacker news UX could also be roughly the same)

Re: Local AI needs to be the norm

#474

Post by reshef316 »

not saying i disagree with the general statement, but there need to be options, not everyone has a machine capable of doing the same type of lifting required to properly run a local version. so what, if my machine is older i'll be locked out? restricted? forced to pay?

Re: Local AI needs to be the norm

#475

Post by manmal »

digitaltrees wrote: post #269
reisse wrote: post #212

Earlier quoted context omitted.

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

I built my own IDE and run my own model specifically to have private agentic coding. I can still access model APIs but I can be purely local if I want too. It’s amazing.

Curious, why did Zed with ACP not work for you?

Re: Local AI needs to be the norm

#476

Post by worldsayshi »

nicce wrote: post #460
Gud wrote: post #437

Earlier quoted context omitted.

The UI is already great. I can’t wait to run my models locally. The sooner I can do my shit without some American mega corp gulping down all my data, the better.

I fear that easier it gets to run models locally, more expensive all the hardware gets. So at the same time it gets further and further. You should have bought the hardware yesterday.

The more expensive it gets, the higher the incentive for more competition in the hardware space.

Re: Local AI needs to be the norm

#478

Post by nicce »

worldsayshi wrote: post #476
nicce wrote: post #460

Earlier quoted context omitted.

I fear that easier it gets to run models locally, more expensive all the hardware gets. So at the same time it gets further and further. You should have bought the hardware yesterday.

The more expensive it gets, the higher the incentive for more competition in the hardware space.

The thing is that it is something which takes so long time. E.g. why Taiwan is still so important?

Re: Local AI needs to be the norm

#479

Post by dust1n »

reisse wrote: post #212
pronik wrote: post #80

They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution"…

> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and ev…

Can you share how you use it to categorize trip photos!

Re: Local AI needs to be the norm

#480

Post by dgb23 »

QuadrupleA wrote: post #374

Not sure how excited I feel about visiting your website and having it auto-download a 8GB model with GPT-3.5 level hallucinations, and then probably crash because I only have 6GB of VRAM. My dad won't be able to use it, or anyone else without a bleeding edge device. On a powerful enough "neural engine" device the battery will be drained quickly, while the heatsink burns a hole in my lap.

Local could also mean self hosted.

The obvious optimization for the case presented would be to generate all the summaries on a server instead of in the client. Then the totally used compute would scale with the number of articles instead of number of users.

Post reply on HN