Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia RTX Spark

nvidia.com

Post reply on HN
Print view View original

121–130 of 437 posts

Re: Nvidia RTX Spark

#121

Post by fg137 »

dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

The best open weight LLMs don't run on this computer, or almost any consumer grade computer. Even the memory requirement for Gemma 4 is out of reach for most consumers (by which I mean those who are not on HN). Unless there is some magic that would make high quality LLMs consume no more than 8GB RAM which makes them usable on a 16GB laptop (which is the norm these days), "local LLM for personal computing" is mostly just a myth.

Re: Nvidia RTX Spark

#123

Post by general_reveal »

comandillos wrote: post #75

So they have basically reused the same hardware as in the DGX Spark (GB10)... That chip isn't great for LLM inference actually. https://www.techpowerup.com/gpu-specs/gb10.c4342 https://www.nvidia.com/en-us/products/rtx-spark/

The RTX GPU laptops run very hot. Even though they are pound for pound better, it’s just runs too hot for local llm usage for me at least. Prefer Macs for this. A lot of AMD cards also run cooler. I wonder if undervting would help with smaller models and heat.

Re: Nvidia RTX Spark

#124

Post by OtherShrezzing »

VMG wrote: post #98
dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

Convince me 1. in order to run LLMs, especially the best ones, you need complicated devices which are expensive 2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot It seems to me that it will always be more economical that the LLM-running devices are in a datacenter where it is easier to make sure they are always utilized

>2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot

I think consumers are primed for that type of behaviour though. I have an iPhone on my desk. It has something like 2-3tflops CPU+GPU, which is double that of the largest super computer on earth when Jurassic Park came out, and is probably more computing power than existed on earth when I was born in the 80s.

I use this device for around 1hr per day to write text messages.

Re: Nvidia RTX Spark

#125

Post by orthoxerox »

Well, it was only a matter of time, since both AMD and now Intel are now switching to APUs. Nvidia could either cede the desktop GPU market to them, going all-in into AI datacenter chips, or it could challenge them.

Maybe the Nth time's the charm and Microsoft+Nvidia will manage to make Windows on ARM a viable platform.

Re: Nvidia RTX Spark

#126

Post by xdertz »

dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

I find it hard to see how that would ever be economical. LLMs need very expensive power hungry chips and datacenters have

- bulk discounts - cheaper electricity - high utilisation to spread the costs among many users

I don't see how PCs could ever compete against it. Most users AI demands would probably result in >90% idle time on the GPU.

Re: Nvidia RTX Spark

#127

Post by Mordisquitos »

VMG wrote: post #98
dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

Convince me 1. in order to run LLMs, especially the best ones, you need complicated devices which are expensive 2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot It seems to me that it will always be more economical that the LLM-running devices are in a datacenter where it is easier to make sure they are always utilized

The trend over the past three decades of personal computing has been for devices to become exponentially more powerful regardless of the actual computing needs of users. The excess computing power has famously been requested by projects such as SETI@Home and Folding@Home, and been exploited by bad actors for crypto mining. The most basic laptop today used only for web browsing and word processing would be a powerful workstation 20 years ago, when the most basic laptop was also used only for web browsing and word processing (and arguably for more things, as it was all mostly local software).

There is no ceiling to the power of consumer hardware. If it's cheap enough, it will be bought.

Re: Nvidia RTX Spark

#128

Post by fg137 »

VMG wrote: post #98
dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

Convince me 1. in order to run LLMs, especially the best ones, you need complicated devices which are expensive 2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot It seems to me that it will always be more economical that the LLM-running devices are in a datacenter where it is easier to make sure they are always utilized

This.

Even two or three years people were pointing out "The ChatGPT subscriptions you can buy with $2000 give you much more compute than whatever home setup you come up with" on r/LocalLLM. I did my own elementary school maths and came to the same conclusion.

Yet till this day people still boast how their beefy M4 Pro/Max machine with 32+GB RAM (which is not at all a "normal person's setup" and costs $2000+) runs LLMs smoothly, and "that's the future".

Someone needs to re-learn basic maths and take a walk around Best Buy to understand what "consumer laptop" looks like.

Re: Nvidia RTX Spark

#129

Post by OtherShrezzing »

VMG wrote: post #98
dom96 wrote: post #92

I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.

Convince me 1. in order to run LLMs, especially the best ones, you need complicated devices which are expensive 2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot It seems to me that it will always be more economical that the LLM-running devices are in a datacenter where it is easier to make sure they are always utilized

If a model is substantially better than most humans at most tasks, the human isn't going to be able to perceive the difference between Claude Opus 7.7 and 8.7. Humans at some point aren't going to be able to perceive the difference on benchmarks either, because they are going to get wildly abstract.

AI vendors are really going to struggle to shift tokens far beyond the frontier of human capabilities. It's reasonable (not guaranteed) to assume that, if the trend of frontier models (doubling capabilities on benchmarks every n months) holds, then the same trend will hold for local models, and those local models will meet and exceed the perception frontier. This would mean a human cannot tell the difference between Mistral-Open-2030 and Claude Opus 2030.

That's a bunch of "ifs", but there's nothing exceptional about those "ifs". They're basically the scenario if nothing changes between now and ~2030 with regards to capabilities trend attainment.

Post reply on HN