Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI is driving the biggest change in laptops in decades

spectrum.ieee.org

Post reply on HN
Print view View original

31–40 of 273 posts

Re: Local AI is driving the biggest change in laptops in decades

#31

Post by sandworm101 »

gguncth wrote: post #22

I have no desire to run an LLM on my laptop when I can run one on a computer the size of six football fields.

I've been playing around with my own home-built AI server for a couple months now. It is so much better than using a cloud provider. It is the difference between drag racing in your own car, and renting one from a dealership. You are going to learn far more doing things yourself. Your tools will be much more consistent and you will walk away with a far greater understanding of every process.

A basic last-generation PC with something like a 3060ti (12GB) is more than enough to get started. My current rig pulls less than 500w with two cards (3060+5060). And, given the current temperature outside, the rig helps heat my home. So I am not contributing to global warming, water consumption, or any other datacenter-related environmental evil.

Re: Local AI is driving the biggest change in laptops in decades

#32

Post by p1esk »

Zambyte wrote: post #26
Legend2440 wrote: post #17

Earlier quoted context omitted.

LLMs are enormously bandwidth hungry. You have to shuffle your 800GB neural network in and out of memory for every token, which can take more time/energy than actually doing the matrix multiplies. GPUs are almost not high bandwidth enough.

This doesn't seem right. Where is it shuffling to and from? My drives aren't fast enough to load the model every token that fast, and I don't have enough system memory to unload models to.

It is right. The shuffling is from CPU memory to GPU memory, and from GPU memory to GPU. If you don’t have enough memory you can’t run the model.

Re: Local AI is driving the biggest change in laptops in decades

#34

Post by zamadatix »

Zambyte wrote: post #26
Legend2440 wrote: post #17

Earlier quoted context omitted.

LLMs are enormously bandwidth hungry. You have to shuffle your 800GB neural network in and out of memory for every token, which can take more time/energy than actually doing the matrix multiplies. GPUs are almost not high bandwidth enough.

This doesn't seem right. Where is it shuffling to and from? My drives aren't fast enough to load the model every token that fast, and I don't have enough system memory to unload models to.

If you're using a MoE model like DeepSeek V3 the full model is 671 GB but only 37 GB are active per token, so it's more like running a 37 GB model from the memory bandwidth perspective. If you do a quant of that it could e.g. be more like 18 GB.

Re: Local AI is driving the biggest change in laptops in decades

#35

Post by marcus_holmes »

fwipsy wrote: post #14

Seems like wishful thinking. > How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly. Why not extrapolate from open-source AIs which are available? The most powerful open-source AI (which I know of) is Kimi K2 and >600gb. Running this at acceptable speed requires 600+gb GPU/NPU memory. Even $2000-3000 AI-focused PCs like the DGX spark or Strix Halo typi…

> In 2010 the typical consumer PC had 2-4gb of RAM. Now the typical PC has 12-16gb. This suggests RAM size doubling perhaps every 5 years at best. If that's the case, we're 25-30 years away from the typical PC having enough RAM to run Kimi K2.

Part of the reason that RAM isn't growing faster is that there's no need for that much RAM at the moment. Technically you can put multiple TB of RAM in your machine, but no-one does that because it's a complete waste of money [0]. Unless you're working in a specialist field 16Gb of RAM is enough, and adding more doesn't make anything noticeably faster.

But given a decent use-case, like running an LLM locally, and you'd find demand for lots more RAM, and that would drive supply, and new technology developments, and in ten years it'll be normal to have 128TB of RAM in a baseline laptop.

Of course, that does require that there is a decent use-case for running an LLM locally, and your point that that is not necessarily true is well-made. I guess we'll find out.

[0] apart from a friend of mine working on crypto who had a desktop Linux box with 4TB of RAM in it.

Re: Local AI is driving the biggest change in laptops in decades

#37

Post by marcus_holmes »

autoexec wrote: post #30
marcus_holmes wrote: post #27

Earlier quoted context omitted.

Doesn't this lead to a lot of tension between the hardware makers and Microsoft? MS wants everyone to run Copilot on their shiny new data centre, so they can collect the data on the way. Laptop manufacturers are making laptops that can run an LLM locally, but there's no point in that unless there's a local LLM to run (and Windows won't have that because Copilot). Are they going to be pre-installing Llama on new lapto…

> MS wants everyone to run Copilot on their shiny new data centre, so they can collect the data on the way. MS doesn't care where your data is, they're happy to go digging through your C drive to collect/mine whatever they want, assuming you can avoid all the dark patterns they use to push you to save everything on OneDrive anyway and they'll record all your interactions with any other AI using Recall

I had assumed that they needed the usage to justify the investment in the data centre, but you could be right and they don't care.

Re: Local AI is driving the biggest change in laptops in decades

#38

Post by zamadatix »

socketcluster wrote: post #13

I feel like there's no point to get a graphics card nowadays. Clearly, graphics cards are optimized for graphics; they just happened to be good for AI but based on the increased significance of AI, I'd be surprised if we don't get more specialized chips and specialized machines just for LLMs. One for LLMs, a different one for stable diffusion. With graphics processing, you need a lot of bandwidth to get stuff in and…

> Clearly, graphics cards are optimized for graphics; they just happened to be good for AI

I feel like the reverse has been true since after the Pascal era.

Re: Local AI is driving the biggest change in laptops in decades

#39

Post by Legend2440 »

Zambyte wrote: post #26
Legend2440 wrote: post #17

Earlier quoted context omitted.

LLMs are enormously bandwidth hungry. You have to shuffle your 800GB neural network in and out of memory for every token, which can take more time/energy than actually doing the matrix multiplies. GPUs are almost not high bandwidth enough.

This doesn't seem right. Where is it shuffling to and from? My drives aren't fast enough to load the model every token that fast, and I don't have enough system memory to unload models to.

From VRAM to the tensor cores and back. On a modern GPU you can have 1-2tb moving around inside the GPU every second.

This is why they use high bandwidth memory for VRAM.

Re: Local AI is driving the biggest change in laptops in decades

#40

Post by zamadatix »

Legend2440 wrote: post #15
Morromist wrote: post #9

I was in the market for a laptop this month. Many new laptops now advertise AI features like this "HP OmniBook 5 Next Gen AI PC" which advertises: "SNAPDRAGON X PLUS PROCESSOR - Achieve more everyday with responsive performance for seamless multitasking with AI tools that enhance productivity and connectivity while providing long battery life" I don't want this garbage on my laptop, especially when its running of its…

Laptop manufacturers are too desperate to cash on the AI craze. There's nothing special about an 'AI PC'. It's just a regular PC with Windows Copilot... which is a standard Windows feature anyway. >I don't want this garbage on my laptop, especially when its running of its battery! The one bit of good news is it's not going to impact your battery life because it doesn't do any on-device processing. It's just calling a…

> It's just a regular PC with Windows Copilot... which is a standard Windows feature anyway.

"AI PC" branded devices get "Copilot+" and additional crap that comes with that due to the NPU. Despite desktops having GPUs with up to 50x more TOPs than the requirement, they don't get all that for some reason https://www.thurrott.com/mobile/copilot-pc/323616/microsoft-...

Post reply on HN