Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The impact of competition and DeepSeek on Nvidia

youtubetranscriptoptimizer.com

Post reply on HN
Print view View original

151–160 of 500 posts

Re: The impact of competition and DeepSeek on Nvidia

#151

Post by colinnordin »

Great article.

>Now, you still want to train the best model you can by cleverly leveraging as much compute as you can and as many trillion tokens of high quality training data as possible, but that's just the beginning of the story in this new world; now, you could easily use incredibly huge amounts of compute just to do inference from these models at a very high level of confidence or when trying to solve extremely tough problems that require "genius level" reasoning to avoid all the potential pitfalls that would lead a regular LLM astray.

I think this is the most interesting part. We always knew a huge fraction of the compute would be on inference rather than training, but it feels like the newest developments is pushing this even further towards inference.

Combine that with the fact that you can run the full R1 (680B) distributed on 3 consumer computers [1].

If most of NVIDIAs moat is in being able to efficiently interconnect thousands of GPUs, what happens when that is only important to a small fraction of the overall AI compute?

[1]: https://x.com/awnihannun/status/1883276535643455790

Re: The impact of competition and DeepSeek on Nvidia

#152

Post by andrewgross »

rahimnathwani wrote: post #86
andrewgross wrote: post #73

Earlier quoted context omitted.

Is there a concept of an expert that persists across layers? I thought each layer was essentially independent in terms of the "experts". I suppose you could look at what part of each layer was most likely to trigger together and segregate those by GPU though. I could be very wrong on how experts work across layers though, I have only done a naive reading on it so far.

I suppose you could look at what part of each layer was most likely to trigger together and segregate those by GPU though Yes, I think that's what they describe in section 3.4 of the V3 paper. Section 2.1.2 talks about "token-to-expert affinity". I think there's a layer which calculates these affinities (between a token and an expert) and then sends the computation to the GPUs with the right experts. This doesn't sou…

Ahh got it, thanks for the pointer. I am surprised there is enough correlation there to allow an entire GPU to be specialized. I'll have to dig in to the paper again.

Re: The impact of competition and DeepSeek on Nvidia

#153

Post by Kubuxu »

andrewgross wrote: post #152
rahimnathwani wrote: post #86

Earlier quoted context omitted.

I suppose you could look at what part of each layer was most likely to trigger together and segregate those by GPU though Yes, I think that's what they describe in section 3.4 of the V3 paper. Section 2.1.2 talks about "token-to-expert affinity". I think there's a layer which calculates these affinities (between a token and an expert) and then sends the computation to the GPUs with the right experts. This doesn't sou…

Ahh got it, thanks for the pointer. I am surprised there is enough correlation there to allow an entire GPU to be specialized. I'll have to dig in to the paper again.

I don't think entire GPU is specialised nor a singular token will use the same expert. I think about it as a gather-scatter operation at each layer.

Let's say you have an inference batch of 128 chats, at layer `i` you take the hidden states, compute their routing, scatter them along with the KV for those layers among GPUs (each one handling different experts), the attention and FF happens on these GPUs (as model params are there) and they get gathered again.

You might be able to avoid the gather by performing the routing on each of the GPUs, but I'm generally guessing here.

Re: The impact of competition and DeepSeek on Nvidia

#154

Post by medler »

adastra22 wrote: post #93
isatty wrote: post #92

Earlier quoted context omitted.

Sorry, I don’t know who George Hotz is, but why isn’t AMD making better drivers for AMD?

George Hotz is a hot Internet celebrity that has basically accomplished nothing of value but has a large cult following. You can safely ignore. (Famous for hacking the PS3–except he just took credit for a separate group’s work. And for making a self-driving car in his garage—except oh wait that didn’t happen either.)

He took an “internship” at Twitter/X with the stated goal of removing the login wall, apparently failing to realize that the wall was a deliberate product decision, not a technical challenge. Now the X login wall is more intrusive than ever.

Re: The impact of competition and DeepSeek on Nvidia

#155

Post by eigenvalue »

nejsjsjsbsb wrote: post #120
punkspider wrote: post #116

Earlier quoted context omitted.

I was excited as soon as I saw the domain name. Even after a few months, this article[1] is still at the top of my mind. You have a certain way of writing. I remember being surprised at first because I thought it would feel like a wall of text. But it was such a good read and I felt I gained so much. 1: https://youtubetranscriptoptimizer.com/blog/02_what_i_learne...

I was put off by the domain by bias against something that sounds like a company blog. Especially a "YouTube something". You may get more milage from excellent writing on a yourname.com. This is a piece that sells you not this product, plus it feels more timeless. In 2050 someone my point to this post. Better if it were on your own name.

I had no idea this would get so much traction. I wanted to enhance my organic search ranking of my niche web app, not crash the global stock market!

Re: The impact of competition and DeepSeek on Nvidia

#156

Post by whizzter »

infecto wrote: post #143
zby wrote: post #101

Earlier quoted context omitted.

" we are going to move it off AMD to our own or partner silicon. We have developed it to be very portable." https://x.com/__tinygrad__/status/1879617702526087346

Honest question. That sounds more difficult that getting things to play with commodity hardware. Maybe I am oversimplifying it though.

They have their own nn,etc libraries so adapting should be fairly focused and AMD drivers have a hilariously bad reputation historically among people who program GPU's (I've been bitten a couple of times myself by weirdness).

I think you should consider it as, if they're trying to avoid Nvidia and make sure their code isn't tied to NVidia-isms, and AMD is troublesome enough for basics the step to customized solutions is small enough to be worthwhile for something even cheaper than AMD.

Re: The impact of competition and DeepSeek on Nvidia

#157

Post by brandonpelfrey »

Great article. I still feel like very few people are viewing the Deepseek effects in the right light. If we are 10x more efficient it's not that we use 1/10th the resources we did before, we expand to have 10x the usage we did before. All technology products have moved this direction. Where there is capacity, we will use it. This argument would not work if we were close to AGI or something and didn't need more, but I don't think we're actually close to that at all.

Re: The impact of competition and DeepSeek on Nvidia

#158

Post by tomrod »

colinnordin wrote: post #151

Great article. > Now, you still want to train the best model you can by cleverly leveraging as much compute as you can and as many trillion tokens of high quality training data as possible, but that's just the beginning of the story in this new world; now, you could easily use incredibly huge amounts of compute just to do inference from these models at a very high level of confidence or when trying to solve extremely…

Conversely, how much larger can you scale if frontier models only currently need 3 consumer computers?

Imagine having 300. Could you build even better models? Is DeepSeek the right team to deliver that, or can OpenAI, Meta, HF, etc. adapt?

Going to be an interesting few months on the market. I think OpenAI lost a LOT in the board fiasco. I am bullish on HF. I anticipate Meta will lose folks to brain drain in response to management equivocation around company values. I don't put much stock into Google or Microsoft's AI capabilities, they are the new IBMs and are no longer innovating except at obvious margins.

Re: The impact of competition and DeepSeek on Nvidia

#159

Post by surajrmal »

csomar wrote: post #131
idonotknowwhy wrote: post #126

Earlier quoted context omitted.

> Unless something radically changed in the last couple years, I am not sure where you got this from? This was the first thing that stuck out to me when I skimmed the article, and the reason I decided to invest the time reading it all. I can tell the author knows his shit and isn't just parroting everyone's praise for AMD Linux drivers. > (I am specifically talking about GPUs for computer usage rather than training/i…

Wayland was a requirement for me. I've used an AMD GPU for years. I had a bug exactly once with a linux update. But has been stable since.

Wayland doesn't matter in the server space though.

Re: The impact of competition and DeepSeek on Nvidia

#160

Post by VHRanger »

brandonpelfrey wrote: post #157

Great article. I still feel like very few people are viewing the Deepseek effects in the right light. If we are 10x more efficient it's not that we use 1/10th the resources we did before, we expand to have 10x the usage we did before. All technology products have moved this direction. Where there is capacity, we will use it. This argument would not work if we were close to AGI or something and didn't need more, but I…

Correct. This effect is known in economics since forever - new technology has

- An "income effect". You use the thing more because it's cheaper - new usecases come up

- A "substitution effect." You use other things more because of the savings.

I got into this on labor economics here [1] - you have counterintuitive examples with ATMs actually increasing the number of bank branches for several decades.

[1]: https://singlelunch.com/2019/10/21/the-economic-effects-of-a...

Post reply on HN