"platform co."
Platform company, as in they're allowing developers on their AI platform and opening an app store?
Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
101–110 of 340 posts
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#102I think at this point, they should stop making it video “cards” but rather video “stations”, a full tower station with power supply and one giant “card” inside with proper cooling, etc., might also justify the crazy prices anyway.
SXM for desktop would be great but it won't happen. The PC industry can't even adopt things like 12VO.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#103Earlier quoted context omitted.
Please do not give them this idea. Ubuntu is actually a pretty great daily driver desktop Linux, and I'd hate for that to lose priority and disappear. I'm not a fan of what happened to the Red Hat ecosystem for exactly the same reasons.
Tbh, Ubuntu’s only pull is the support and breadth of users. As a desktop, it’s let down by Unity, which IMHO is basically a port of Windows 8 tablet UI. If they defaulted back to a menu and taskbar-based WM, it might actually be more approachable to users who are more familiar with macOS and Windows.
Main Ubuntu hasn't shipped with Unity for like 7 years.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#104I haven't listened to Jensen speak before, but am I the only one who thought the presentation wasn't very polished? Not a knock on anything he has accomplished, just an observation that sorta surprised me
I think it's a good reminder that objectively great CEO's and leaders can be kind of cringe when presenting. A lot of times people like that get passed up in promotions in favor of smooth talkers.
It's been said that founders are people who can't get hired.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#105Platform co seems fitting, considering Nvidia's data center revenue in the fourth quarter of 2023 was a record $18.4 billion, which is 27% higher than the previous quarter and 409% higher than the previous year.
Seems revenue from inference is growing at a significant clip.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#106FP8 being 2.5x Hopper is kind of disappointing after such a long time. Since its 2 fused chips, that means it’s 25% effective delta. though it seems most of the progress has been on memory throughput and power use which is still very impressive. I wonder how this will trickle down to the consumer segment.
Jensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.
The other big announcement here is NIM - Nvidia Inference Microservice.
It's basically TensorRT-LLM + Triton Inference Server + pre-build of models to TensorRT-LLM engines + packaging + what appears to be an OpenAI compatible API router in front of all of it + other "enterprise" management and deployment tools.
This software stack is extremely performant and very flexible, I've noted here before it's what many large-scale hosted inference providers are already using (Amazon, Cloudflare, Mistral, etc).
From the article:
'Nvidia will work with AI companies like Microsoft or Hugging Face to ensure their AI models are tuned to run on all compatible Nvidia chips. Then, using a NIM, developers can efficiently run the model on their own servers or cloud-based Nvidia servers without a lengthy configuration process.
“In my code, where I was calling into OpenAI, I will replace one line of code to point it to this NIM that I got from Nvidia instead,” Das said.'
The dead giveaway is "I changed one line of code in my OpenAI code" which means "I pointed the OpenAI API base URL to an OpenAI compatible API proxy that likely interfaces with Triton on the backend via its gRPC protocol".
I have a lot of experience with TensorRT-LLM + Triton and have been working on a highly performant rust-based open source project for the OpenAI compatible API and routing portion[0].
On this hardware (FP4) with this software package 30x compared to other solutions (who knows what - base transformers?) on Hopper seems possible. TensorRT-LLM and Triton can already do FP8 on Hopper and as noted the performance is impressive.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#107"platform co." Platform company, as in they're allowing developers on their AI platform and opening an app store?
The headline is editorialized by the submitter, the actual headline is “Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’” which is arguably worse.
The article doesn’t discuss becoming a platform co but instead discussed ways their existing platform subscription model is evolving to add backwards compatibility testing.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#108"Platform company" means multi-chip in this case? Seems logical since it's becoming impractical to cram so many transistors on a single die.
I don't really understand the bird's-eye view of the product line, but judging by some of the raw physical numbers and configurations Jensen was bragging about, it means that they want to basically play the mainframe game of locking high-end applications into proprietary middleware running on proprietary chassis with proprietary cluster interconnect (hello, Mellanox acquisiton).
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#109Double digit peta flop mass produced.
"The computing power needed to replicate the human brain’s relevant activities has been estimated by various authors, with answers ranging from 10^12 to 10^28 FLOPS."
Petaflop is 10^15
Crazy times.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#110"Platform company" means multi-chip in this case? Seems logical since it's becoming impractical to cram so many transistors on a single die.
It means all the main chips required for a large-scale datacenter. And many of the layers of software on top of it.
Hardware: * The GPU * The GPU-GPU Fabric (NVLINK) * The CPU * The NIC * The Network Fabric (infiniband) * The Switch
And that's not even starting to get into the many layers of the software stack (CUDA, Riva, Megatron, Omniverse) that they're contributing and working to get folks to build on.