Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

731–740 of 804 posts

Re: Local AI needs to be the norm

#731

Post by DoctorOetker »

DrScientist wrote: post #515
DoctorOetker wrote: post #334

One advantage of local AI is continual learning. When I say 'moat' I don't mean moat specific to a company vis-a-vis other companies, but 'moat' specific to the set of inference providers vis-a-vis self-hosted local inference. The moat consists primarily of being able to batch inference requests. If we pretend people weren't interested in long context-lengths, there would be a moat for inference providers. who can ba…

Anybody know of good real world examples for continual learning? Does it really work?

In this case I think you'd want to use Source-Aware-Training [0] to associate a "timestamp" vector to each native context chunk (perhaps overlapping) of conversation, probably the weights using a kind of Gray code so that the LLM has the immediate out-of-native context history can be retrieved through the nearby gray code of 1, 2, etc steps ago compared to the current timestep gray code.

https://arxiv.org/abs/2404.01019

Re: Local AI needs to be the norm

#732

Post by voncheese »

DrewADesign wrote: post #664
voncheese wrote: post #619

Earlier quoted context omitted.

As long as Apple and Google put reasonable AI capabilities on device, then software engineers will use those capabilities when it makes sense (the article gives lots of good examples of capabilities that make sense to run locally). As the author notes, it's cheaper and more reliable to run these things locally. That also doesn't preclude LLM services from being massively successful, they'll just have to justify the p…

> That also doesn't preclude LLM services from being massively successful, they'll just have to justify the pricing and complexity that comes with their adoption, just like any other product. What is completely different from every other product is how much they’re spending, and how much they’re obligating themselves to spend going forward. I think there’s a very good chance that the existing providers could be miles…

Yeah that's a good callout for sure, the spending here is nuts so agree that it's not "just another business that has to price itself right to be competitive".

I guess if the time horizons is long, like 20 years, then maybe the spending, as it begins to amortize, gets more in line?

I was thinking that a comparison could be to cloud providers, each of which had to spend a lot of money to build out datacenter before making money. Difference there is AWS proved the product first, so when Microsoft and Google came along, they knew it would work and be profitable. With AI, nobody has proven it will work and be profitable, they're all competing for that at the same time which is a potentially dangerous mix for the reasons you cited.

Re: Local AI needs to be the norm

#733

Post by kelnos »

driese wrote: post #417
emadb wrote: post #393

Earlier quoted context omitted.

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

For the same reason that a human who is fluent in five languages can probably express themselves better in either one compared to human that only speaks one, while also having a more nuanced understanding of general grammar. From what I know, learning on a more diverse set makes a model better overall.

Humans brains and LLMs are not the same, though. I don't think your analogy is remotely applicable, even if your conclusion may be correct.

Re: Local AI needs to be the norm

#734

Post by coevcan »

chakintosh wrote: post #624

I'm literally working on an iOS app right now that needs to infer some input fields from free text typed by the user. Now to take into consideration typos, unstructured text (pricing, dates .. etc), I was pondering a cloud LLM or a basic local parser or even a local on-device LLM (ANE for 15+ devices and a different on-device LLM for the older models) For the different on-device LLM, I literally went to HuggingFace a…

Apple includes a local LLM on all recent iPhones, https://developer.apple.com/documentation/foundationmodels. Seems like a bad idea to force your users to download a 3GB LLM just to parse a text field.

Re: Local AI needs to be the norm

#735

Post by binyu »

segmondy wrote: post #723
binyu wrote: post #311

Earlier quoted context omitted.

They all still fall short of Opus 4.6, definitely though. They are good but fail on extremely complex tasks, in contrast with a frontier model that will keep on trying until it succeeds or exhausts the solutions space.

frontier models don't keep trying until they succeed. that's a harness problem and best believe it, the best harness are private and not public.

It is much more of a context window size and model capabilities problem. Local models are not even remotely close in solving complex problems, even when used with the same harness.

Re: Local AI needs to be the norm

#736

Post by zozbot234 »

hedora wrote: post #727
Fokamul wrote: post #510

Earlier quoted context omitted.

I'm just guessing, but IDE which is using 3D acceleration just for stupid UI to run "smoothly", that is ridiculous. Who runs IDE with LLM agents accessing your local filesystem, on bare metal? Or am I alone to run everything LLM related on my VM just for development work. Then because of ZED genius decision, you need to share your GPU to VM, then some important features will not work, like snapshots. So you also need…

What's wrong with using a 3d accelerator and falling back to CPU graphics if needed? Pixels / joule is orders of magnitude better on an iGPU than on the CPU. (Which can matter over a 8-12 hour editing session, maybe.)

Modern IDEs don't use 3D at all, nor do they use the sprite-like 2D graphics that GPUs excel at and that can accelerate, e.g. mobile touch- and swipe-based UX. The main thing they do is font rendering, and accelerating that on GPU while keeping visual quality unchanged is quite complicated. The graphics pipeline doesn't really help all that much.

Re: Local AI needs to be the norm

#737

Post by crazygringo »

themafia wrote: post #708
crazygringo wrote: post #701

Earlier quoted context omitted.

You literally wrote: > I've never had a "What's new" tab ever open because I disable the customized home page where that's displayed. I'm not "denying your experience" of not having the what's new tab. I'm denying your explanation for it. You wrongly thought it was due to disabling the home page, and then you were insulting to the parent with the snarky "I'm guessing you're not aware that's an option". You were the o…

> and then you were insulting to the parent with the snarky And what did they say to me? "I'm guessing you immediately close the What's New Chrome tab when you update?" If that's not snarky then I don't know what is. It's rude, churlish, and presumptive. Yet you take absolutely no notice of this as if you didn't care to do anything other than attack me. > You wrongly thought it was due to disabling the home page And…

> And what did they say to me?

It doesn't matter. It's not a reason to be insulting back. I'm not insulting you here, even though you're insulting me.

> And you are 100% certain it isn't? Why?

I already explained. This is trivial to test. Another commenter here also provided the more likely explanation.

> If it's so inappropriate then why did you reply?

To let you know why, so you can learn from this. When people are insulting to me, I don't just ignore it so people can walk all over me. But I also don't insult back. I explain what they're doing so they can learn to have better manners.

It's not picking "one small nit", it's standing up to abusive language and behavior ("corporate defense squad members", "gaslighting nonsense"). Please be better in the future.

Re: Local AI needs to be the norm

#738

Post by zozbot234 »

reissbaker wrote: post #721
zozbot234 wrote: post #636

Earlier quoted context omitted.

But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.

The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies…

> The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models.

Sometimes it really is free though, because the hardware was bought to serve some other existing needs and that capital expense was fully depreciated quite some time ago. Underutilised hardware is essentially ubiquitous.

> Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig.

But using that 8xB200 setup to run inference on cheap, non-frontier models is a plain waste. Its highest and best use is in an AI datacenter serving exceptionally smart models like Gemini DeepThink, GPT Pro or Claude Mythos. (If this isn't true, it means that the current level of large-scale investment in frontier, super intelligent AI is misplaced, and you should worry about that; not whether some models are best ran on lower-end hardware!)

Re: Local AI needs to be the norm

#739

Post by degamad »

amelius wrote: post #458
thot_experiment wrote: post #451

Earlier quoted context omitted.

False. The absolute capability is irrelevant, with the proper harness 31b is more than adequate for a very large portion of the tasks I ask AI to do. The metric isn't how good the model is at Erdos Problems, it's how reliably it can remove drudgery in my life. It just autonomously reverse engineered a bluetooth protocol with minimal intervention, it's ability to react to data and ground itself is constantly impressiv…

This is like saying that 640kB is enough for anybody.

It's more like saying "HIMEM.SYS is not much better than 640kB".

Re: Local AI needs to be the norm

#740

Post by alfiedotwtf »

zozbot234 wrote: post #706
alfiedotwtf wrote: post #685

Earlier quoted context omitted.

I've got a 4060 Ti 12Gb with 128Gb RAM. I was hoping once I could demonstrate to myself that I could run Deepseek v4 Flash locally (even at really slow speeds), then it would be worth my time and money to get something to run it > 20t/s. ... currently testing out Stepfun 3.5 Flash Q4_k_m as a stop gap (unless it blows my socks off first).

I don't think the DS4 project supports the CPU/GPU split approach you'd need for best performance on that kind of hardware (shared layers on GPU, most experts on CPU). CPU-only inference would work but might be slow.

Ah dang. Hmm, damn this hobby is expensive. Maybe I should just take up drugs instead
Post reply on HN