Seems Nvidia is going for maximum margin as they see competition ahead.
[deleted]
Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
61–70 of 340 posts
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#62Earlier quoted context omitted.
Jensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.
He always does that. They stack up a bunch of special case features like sparsity that most people don't use in practice to get these unrealistic numbers. It'll be faster, certainly, but 30x will only be achievable in very special cases I'm sure.
Isn't sparsity almost always a win at this point? Making everything fully connected is a major waste.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#63I haven't listened to Jensen speak before, but am I the only one who thought the presentation wasn't very polished? Not a knock on anything he has accomplished, just an observation that sorta surprised me
I've been watching his keynotes for as long as I can remember, this is how it's always been
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#64What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#65What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698
yes
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#66I think at this point, they should stop making it video “cards” but rather video “stations”, a full tower station with power supply and one giant “card” inside with proper cooling, etc., might also justify the crazy prices anyway.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#67Earlier quoted context omitted.
30x is the type of number that when you see it in a generational improvement, you should ignore it as marketing fluff.
From how I understood it, it means they optimised the entire stack from CUDA to the networking interconnects specifically for data centers, meaning you get 30x more inference per dollar for a datacenter. This is probably not fluff, but it's only relevant for a very very specific use-case, ie enterprises with the money to buy a stack to serve thousands of users with LLMs. It doesn't matter for anyone who's not microso…
They showed 30x was for FP4. Who is using FP4 in practice?
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#68Seems Nvidia is going for maximum margin as they see competition ahead.
And they can build a big moat with cuda.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#69Earlier quoted context omitted.
They are priced as if they are the only ones who are capable of creating chips that can crunch LLM algos. But AMD, Google, Intel, and even Apple are also capable. Apple is in talks with Google to bring Gemini to the iPhone, and it will obviously also be on android phones. So almost every phone on earth is poised to be using Gemini in the near future, and Gemini runs entirely on Google's own custom hardware (which is…
This seems as good a place as any to be Corrected by the Internet, so... correct me if I'm wrong. Making a graphics chip that is as good as Nvidia: Very difficult. Huge moat, huge effort, lots of barriers, lots of APIs, lot of experience, lots of decades of experience to overcome. Making something that can run a NN: Much, much easier. I'd guess, start-up level feasible. The math is much simpler. There's a lot of it,…
CUDA is/was their biggest advantage to be honest, not the HW. They saw the demand to super high-end GPUs driven by Bitcoin mining craze thanks to CUDA, and it transitioned gracefully to AI/ML workloads. Google was much more ahead to see the need and develop TPUs for example.
I don't think they have a crazy advantage HW wise. Couple of start-ups are able to achieve this. If SW infrastracture end is standardized, we will have a more level playground.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#70Earlier quoted context omitted.
This seems as good a place as any to be Corrected by the Internet, so... correct me if I'm wrong. Making a graphics chip that is as good as Nvidia: Very difficult. Huge moat, huge effort, lots of barriers, lots of APIs, lot of experience, lots of decades of experience to overcome. Making something that can run a NN: Much, much easier. I'd guess, start-up level feasible. The math is much simpler. There's a lot of it,…
I agree with you, but let me devil's advocate. After 10 years of pretending to care about compute, AMD has filled the industry with burned-once experts who, when weighing nvidia against competitors, instinctively include "likely boondoggle" against every competitor's quote because they've seen it happen, possibly several times. Combine this with nvidia's deep experience and and huge rich-get-richer R&D budget keeping…
> burned-once experts
More like burned 2x / 3x / 4x of this time it's different people.
Looking at you Intel