Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI is driving the biggest change in laptops in decades

spectrum.ieee.org

Post reply on HN
Print view View original

21–30 of 273 posts

Re: Local AI is driving the biggest change in laptops in decades

#21

Post by autoexec »

socketcluster wrote: post #13

I feel like there's no point to get a graphics card nowadays. Clearly, graphics cards are optimized for graphics; they just happened to be good for AI but based on the increased significance of AI, I'd be surprised if we don't get more specialized chips and specialized machines just for LLMs. One for LLMs, a different one for stable diffusion. With graphics processing, you need a lot of bandwidth to get stuff in and…

I don't doubt that there will be specialized chips that make AI easier, but they'll be more expensive than the graphics cards sold to consumers which means that a lot of companies will just go with graphics cards, either because the extra speed of specialized chips won't be worth the cost, or will they'll be flat out too expensive and priced for the small number of massive spenders who'll shell out insane amounts of money for any/every advantage (whatever they think that means) they can get over everyone else.

Re: Local AI is driving the biggest change in laptops in decades

#23

Post by sandworm101 »

autoexec wrote: post #20
Legend2440 wrote: post #15

Earlier quoted context omitted.

Laptop manufacturers are too desperate to cash on the AI craze. There's nothing special about an 'AI PC'. It's just a regular PC with Windows Copilot... which is a standard Windows feature anyway. >I don't want this garbage on my laptop, especially when its running of its battery! The one bit of good news is it's not going to impact your battery life because it doesn't do any on-device processing. It's just calling a…

Even collecting and sending all that data to the cloud is going to drain battery life. I'd really rather my devices only do what I ask them to than have AI running the background all the time trying to be helpful or just silently collecting data.

>> I'd really rather my devices only do what I ask them to

Linux hears your cry. You have a choice. Make it.

Re: Local AI is driving the biggest change in laptops in decades

#24

Post by sipjca »

fwipsy wrote: post #14

Seems like wishful thinking. > How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly. Why not extrapolate from open-source AIs which are available? The most powerful open-source AI (which I know of) is Kimi K2 and >600gb. Running this at acceptable speed requires 600+gb GPU/NPU memory. Even $2000-3000 AI-focused PCs like the DGX spark or Strix Halo typi…

> It may be possible to run a "good enough" model on consumer hardware eventually

10-15 years?!!!! What is the definition of good enough? Qwen3 8B or A30B are quite capable models which run on a lot of hardware even today. SOTA is not just getting bigger, it's also getting more intelligence and running it more efficiently. There have been massive gains in intelligence at the smaller model sizes. It is just highly task dependent. Arguably some of these models are "good enough" already, and the level of intelligence and instruction following is much better from even 1 year ago. Sure not Opus 4.5 level, but still much could be done without that level of intelligence.

Re: Local AI is driving the biggest change in laptops in decades

#25

Post by epicureanideal »

fwipsy wrote: post #14

Seems like wishful thinking. > How many TOPS do you need to run state-of-the-art models with hundreds of millions of parameters? No one knows exactly. Why not extrapolate from open-source AIs which are available? The most powerful open-source AI (which I know of) is Kimi K2 and >600gb. Running this at acceptable speed requires 600+gb GPU/NPU memory. Even $2000-3000 AI-focused PCs like the DGX spark or Strix Halo typi…

You may be correct, but I wonder if we'll see Mac Mini sized external AI boxes that do have the 1TB of RAM and other hardware for running local models.

Maybe 100% of computer users wouldn't have one, but maybe 10-20% of power users would, including programmers who want to keep their personal code out of the training set, and so on.

I would not be surprised though if some consumer application made it desirable for each individual, or each family, to have local AI compute.

It's interesting to note that everyone owns their own computer, even though a personal computer sits idle half the day, and many personal computers hardly ever run at 80% of their CPU capacity. So the inefficiency of owning a personal AI server may not be as much of a barrier as it would seem.

Re: Local AI is driving the biggest change in laptops in decades

#26

Post by Zambyte »

Legend2440 wrote: post #17
socketcluster wrote: post #13

I feel like there's no point to get a graphics card nowadays. Clearly, graphics cards are optimized for graphics; they just happened to be good for AI but based on the increased significance of AI, I'd be surprised if we don't get more specialized chips and specialized machines just for LLMs. One for LLMs, a different one for stable diffusion. With graphics processing, you need a lot of bandwidth to get stuff in and…

LLMs are enormously bandwidth hungry. You have to shuffle your 800GB neural network in and out of memory for every token, which can take more time/energy than actually doing the matrix multiplies. GPUs are almost not high bandwidth enough.

This doesn't seem right. Where is it shuffling to and from? My drives aren't fast enough to load the model every token that fast, and I don't have enough system memory to unload models to.

Re: Local AI is driving the biggest change in laptops in decades

#27

Post by marcus_holmes »

Legend2440 wrote: post #15
Morromist wrote: post #9

I was in the market for a laptop this month. Many new laptops now advertise AI features like this "HP OmniBook 5 Next Gen AI PC" which advertises: "SNAPDRAGON X PLUS PROCESSOR - Achieve more everyday with responsive performance for seamless multitasking with AI tools that enhance productivity and connectivity while providing long battery life" I don't want this garbage on my laptop, especially when its running of its…

Laptop manufacturers are too desperate to cash on the AI craze. There's nothing special about an 'AI PC'. It's just a regular PC with Windows Copilot... which is a standard Windows feature anyway. >I don't want this garbage on my laptop, especially when its running of its battery! The one bit of good news is it's not going to impact your battery life because it doesn't do any on-device processing. It's just calling a…

Doesn't this lead to a lot of tension between the hardware makers and Microsoft?

MS wants everyone to run Copilot on their shiny new data centre, so they can collect the data on the way.

Laptop manufacturers are making laptops that can run an LLM locally, but there's no point in that unless there's a local LLM to run (and Windows won't have that because Copilot). Are they going to be pre-installing Llama on new laptops?

Are we going to see a new power user / normal user split? Where power users buy laptops with LLMs installed, that can run them, and normal folks buy something that can call Copilot?

Any ideas?

Re: Local AI is driving the biggest change in laptops in decades

#28

Post by autoexec »

fallat wrote: post #11
noosphr wrote: post #6

Earlier quoted context omitted.

Feast and famine. In three years we will be swimming in more ram than we know what to do with.

Kind of feel that's already the case today... 4GB I find is still plenty for even business workloads.

Video games have driven the need for hardware more than office work. Sadly games are already being scaled back and more time is being spent on optimization instead of content since consumers can't be expected to have the kind of RAM available they normally would and everyone will be forced to make do with whatever RAM they have for a long time.

Re: Local AI is driving the biggest change in laptops in decades

#29

Post by smallerize »

Zambyte wrote: post #26
Legend2440 wrote: post #17

Earlier quoted context omitted.

LLMs are enormously bandwidth hungry. You have to shuffle your 800GB neural network in and out of memory for every token, which can take more time/energy than actually doing the matrix multiplies. GPUs are almost not high bandwidth enough.

This doesn't seem right. Where is it shuffling to and from? My drives aren't fast enough to load the model every token that fast, and I don't have enough system memory to unload models to.

You're probably not using an 800GB model.

Re: Local AI is driving the biggest change in laptops in decades

#30

Post by autoexec »

marcus_holmes wrote: post #27
Legend2440 wrote: post #15

Earlier quoted context omitted.

Laptop manufacturers are too desperate to cash on the AI craze. There's nothing special about an 'AI PC'. It's just a regular PC with Windows Copilot... which is a standard Windows feature anyway. >I don't want this garbage on my laptop, especially when its running of its battery! The one bit of good news is it's not going to impact your battery life because it doesn't do any on-device processing. It's just calling a…

Doesn't this lead to a lot of tension between the hardware makers and Microsoft? MS wants everyone to run Copilot on their shiny new data centre, so they can collect the data on the way. Laptop manufacturers are making laptops that can run an LLM locally, but there's no point in that unless there's a local LLM to run (and Windows won't have that because Copilot). Are they going to be pre-installing Llama on new lapto…

> MS wants everyone to run Copilot on their shiny new data centre, so they can collect the data on the way.

MS doesn't care where your data is, they're happy to go digging through your C drive to collect/mine whatever they want, assuming you can avoid all the dark patterns they use to push you to save everything on OneDrive anyway and they'll record all your interactions with any other AI using Recall

Post reply on HN