Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

701–710 of 804 posts

Re: Local AI needs to be the norm

#701

Post by crazygringo »

themafia wrote: post #494
crazygringo wrote: post #148

Earlier quoted context omitted.

I use an extension that gives me a customized homepage, but I still always get the "what's new" tab on every major version upgrade. It's a totally separate tab that opens. It's got nothing to do with what you use as your homepage.

Thank you for going out of your way to deny my exact experience. Do you think I'm doing this to rag on Google? And you're this eager to defend them? I'm on gentoo. I have to update chrome manually. I updated it. On update I _never_ get a "what's new" page. I've had this profile for more than a decade so I have no actual idea why, but, I can absolutely tell you, I do *not* get one. After update it started consuming al…

You literally wrote:

> I've never had a "What's new" tab ever open because I disable the customized home page where that's displayed.

I'm not "denying your experience" of not having the what's new tab. I'm denying your explanation for it.

You wrongly thought it was due to disabling the home page, and then you were insulting to the parent with the snarky "I'm guessing you're not aware that's an option".

You were the one who wasn't aware of the real explanation. Now you make up a totally unwarranted accusations ("going out of your way to deny my exact experience", "gaslighting nonsense"), and add character assassination on top ("you're this eager to defend them?", "corporate defense squad members").

Your comment is extremely inappropriate. Please re-read the HN guidelines, especially:

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

Re: Local AI needs to be the norm

#702

Post by ToucanLoucan »

DrewADesign wrote: post #603
Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

And this is exactly what the LLM provider industry is fighting tooth-and-nail. It’s not only because it doesn’t directly contribute to their bottom line, it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers. What we’re headed towards would have been a killer product and probably still shifted a bunch of capital to the bazillionaires ha…

> ...it also directly opposes the idea that LLMs are going to replace entire workers rather than enhance the abilities of individual workers.

Which also, as I feel the need to remind everyone every time it comes up, has not yet once been actually shown to be a workable strategy. For any worker in any industry.

And to be clear, I'm talking about a worker, sitting in a chair, replaced with an agent, sitting in... a server, I guess, where nothing else about the org has to be changed. That's what's being advertised and sold, and it has never to my knowledge actually happened.

Re: Local AI needs to be the norm

#703

Post by manmal »

Fokamul wrote: post #510
manmal wrote: post #475

Earlier quoted context omitted.

Curious, why did Zed with ACP not work for you?

I'm just guessing, but IDE which is using 3D acceleration just for stupid UI to run "smoothly", that is ridiculous. Who runs IDE with LLM agents accessing your local filesystem, on bare metal? Or am I alone to run everything LLM related on my VM just for development work. Then because of ZED genius decision, you need to share your GPU to VM, then some important features will not work, like snapshots. So you also need…

Agents are read-only per default in Zed. You should really get off your high horse.

Re: Local AI needs to be the norm

#704

Post by SilentM68 »

t_mahmood wrote: post #420
SilentM68 wrote: post #377

Earlier quoted context omitted.

Not sure if this is exactly the scenario you envision but I run ComfyUI on an Acer Helio 300 laptop, from four years ago. Has 16GB RAM, NVIDIA GeForce RTX 2060 w/6144MiB of VRAM and have generated a few images using "NetaYumev35_pretrained_all_in_one.safetensors" @ 10.6GB checkpoint, (well beyond the 6GB capacity of the RTX 2060 card). That being said, it takes more than 10 minutes to complete the task. Of course, I…

I'm not running local for exactly the same reason, to not stress my components. As it seems we are in for a long haul due to this AI bubble (can't wait for it to pop) so need to make sure I survive this madness, as for sure I can't afford to replace anything right now.

I don't know that any AI bubble will pop. AI can be used to accelerate therapies, cures, make scientific advancements. Add to that, quantum science technology which if successful, should accelerate things, depending on who's the one at the wheel. Problem is the gap between now and then (e.g. age abundance). It's going to be a difficult road for good number of the population until that day comes. I'm scouting potential locations of bridges, to live under, so that I can find and claim one when homeless day arrives.

I can't help but feel that companies using AI, engaging in employee layoffs, are shooting themselves in the foot. The endgame for them will be zero profits, since displaced workers translates to no money to pay for goods and services :|

Re: Local AI needs to be the norm

#705

Post by CamperBob2 »

dleslie wrote: post #695
CamperBob2 wrote: post #255

Earlier quoted context omitted.

Not every country is in a crypto-libertarian race to hoard power and wealth. Meanwhile, in the EU, the model would be collectively financed, trained by a competent, neutral agency... and then completely lobotomized in the name of "the children," "safety," "IP rights," "correct speech," dozens of individual countries' legal and regulatory requirements, and any number of additional vocal, noncontributing NGOs. So no on…

European models are competitive, despite the concerns you raise. I don't need a model that can easily produce CSAM or reproduce copyrighted works verbatim in order to be productive.

EU models are not, in fact, competitive with US or Chinese models.

And that's not how these things work. If you censor the model for one purpose, you will degrade it for others. We both know that the bureaucrats won't stop at either of those purposes. It's not in a censor's nature to walk away satisfied.

Re: Local AI needs to be the norm

#706

Post by zozbot234 »

alfiedotwtf wrote: post #685
zozbot234 wrote: post #668

Earlier quoted context omitted.

What kind of hardware are you planning to run this on? As mentioned already, I've been trying to understand how gracefully it might degrade on 64GB RAM or perhaps lower (the total weights size is 80GB at the provided quant) using SSD offload for the weights, and then (assuming it works and doesn't just OOM) whether the tok/s figures might meaningfully improve in that scenario by running multiple sessions in parallel.

I've got a 4060 Ti 12Gb with 128Gb RAM. I was hoping once I could demonstrate to myself that I could run Deepseek v4 Flash locally (even at really slow speeds), then it would be worth my time and money to get something to run it > 20t/s. ... currently testing out Stepfun 3.5 Flash Q4_k_m as a stop gap (unless it blows my socks off first).

I don't think the DS4 project supports the CPU/GPU split approach you'd need for best performance on that kind of hardware (shared layers on GPU, most experts on CPU). CPU-only inference would work but might be slow.

Re: Local AI needs to be the norm

#707

Post by zozbot234 »

alfiedotwtf wrote: post #686
doctorpangloss wrote: post #665

Earlier quoted context omitted.

you can run it today with mlx if you have 256g or 512g mac studio. no "antirez" fork needed. it isn't that large of a model and the compressed kv implementation is not that complicated the problem is that they released the model in a quantized format that is more complex than it appears, and people make a lot of mistakes working with it. it is quantization-aware-trained, so you can't "just" upscale it and scale down.…

Unfortunately I didn't get a Mac with big ram at the time it was cheap, and I'd personally focus on moving away from Apple and going Linux fulltime at work and home (currently Macbook for laptop connected to my big rig, well it's not that big compared to the AI people in here).

What kind of RAM does your MacBook have? It might still be worth experimenting w/ DS4 using disk offload, though it would be dog slow at best and the RAM would be much too limited for meaningful parallelism, especially for larger contexts.

Re: Local AI needs to be the norm

#708

Post by themafia »

crazygringo wrote: post #701
themafia wrote: post #494

Earlier quoted context omitted.

Thank you for going out of your way to deny my exact experience. Do you think I'm doing this to rag on Google? And you're this eager to defend them? I'm on gentoo. I have to update chrome manually. I updated it. On update I _never_ get a "what's new" page. I've had this profile for more than a decade so I have no actual idea why, but, I can absolutely tell you, I do *not* get one. After update it started consuming al…

You literally wrote: > I've never had a "What's new" tab ever open because I disable the customized home page where that's displayed. I'm not "denying your experience" of not having the what's new tab. I'm denying your explanation for it. You wrongly thought it was due to disabling the home page, and then you were insulting to the parent with the snarky "I'm guessing you're not aware that's an option". You were the o…

> and then you were insulting to the parent with the snarky

And what did they say to me?

"I'm guessing you immediately close the What's New Chrome tab when you update?"

If that's not snarky then I don't know what is. It's rude, churlish, and presumptive. Yet you take absolutely no notice of this as if you didn't care to do anything other than attack me.

> You wrongly thought it was due to disabling the home page

And you are 100% certain it isn't? Why?

> Your comment is extremely inappropriate.

I think your definitions of "snarky" and "extremely" need to be adjusted. You've downvoted me, you disagree, I get that, what more do you want? You've persisted this conversation as if there is more to get from it but all you seem interested in is this off topic browbeating.

Instead of responding to the point, or engaging with it on any level, you've picked one small nit, and then attempted to derail the entire point with it. What /you/ are doing is inappropriate.

If it's so inappropriate then why did you reply? You can't have it both ways.

Re: Local AI needs to be the norm

#709

Post by zozbot234 »

pianopatrick wrote: post #384
winocm wrote: post #247

Earlier quoted context omitted.

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

Currently I'm testing something like this just to see what happens. I have an old laptop with 4GB of RAM. I attached a USB drive with Gemma 4 31B model (which is 32.6 GB). Currently the laptop is running llama.cpp and trying to respond to a prompt by streaming the model from disk. The USB drive light is flickering, showing something is happening. It's been about 8 hours since I entered the prompt and I've gotten abou…

Wow, that's a true worst case scenario especially if the USB is just plain old USB 2.0 (max 480 Mbps) and/or if the drive is a spinning disk. How's the CPU doing, though? Is there any headroom given the USB bottleneck?

Re: Local AI needs to be the norm

#710

Post by tedzhu »

mattlondon wrote: post #47

Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference. Damned if they do, damned if they don't.

Typical HN arguing they need a button to opt in. In reality 99% people don't care, if it works they're fine with that.
Post reply on HN