Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

371–380 of 604 posts

Re: We were wrong about GPUs

#371

Post by transpute »

bonzini wrote: post #364
transpute wrote: post #361

Earlier quoted context omitted.

> Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? Good question for a lawyer. Even more reason (beyond maintenance cost) that it would be best done by Nvidia. qCUDA paper has a couple dozen references on API remoting research, https://www.cs.nthu.edu.tw/~ychung/conference/2019-CloudCom.... > If it has been contributed to QEMU, it isn't GPL/LGPL? Not contributed, but integrated with QEMU…

NVIDIA support is not special as far as QEMU is concerned—the special parts are all in their proprietary device driver, and they talk to QEMU via the VFIO infrastructure for userspace drivers. They just reimplemented the same thing in Cloud Hypervisor. Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.

> NVIDIA.. just reimplemented the same thing in Cloud Hypervisor.

Was that recent, with MIG support for GPGPU partitioning? Is there a public mailing list thread or patch series for that work?

Nvidia has a 90-page deployment doc on vCS ("virtual compute server") for RedHat KVM, https://images.nvidia.com/content/Solutions/data-center/depl...

Re: We were wrong about GPUs

#372

Post by black3r »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

All professional developers want two things: Do their work as fast as possible and spend as little budget to make things work. That's the core operating principle of most companies.

What's changing is that managed solutions are becoming increasingly easier to set up and increasingly cheaper on smaller scales.

While I do personally enjoy understanding the entire stack, I can't justify self-hosting and managing an LLM until we run so many prompts a day that it becomes cheaper for us to run our own GPUs compared to just running APIs like OpenAI/Anthropic/Deepseek/...

Re: We were wrong about GPUs

#373

Post by huntaub »

ozgune wrote: post #366
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

I agree with the blog post that using K8s + containers for GPU virtualization is a security disaster waiting to happen. Even if you configure your container right (which is extremely hard to do), you don't get seccomp-bpf. People started using K8s for training, where you already had a network isolated cluster. Extending the K8s+container pattern to multi-tenant environments is scary at best. I didn't understand the f…

This has actually brought up an interesting point. Kubernetes is nothing more than an API interface. Should someone be working on building a multi-tenant Kubernetes (so that customers don't need to manage nodes or clusters) which enforces VM-level security (obviously you cannot safely co-locate multiple tenants containers on the same VM)?

Re: We were wrong about GPUs

#374

Post by _heimdall »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

I agree that split exists, and that the former is more rare, but in my experience the split is less about avoid magic and more about keeping control of your system.

Many, likely most, developers today don't care about controlling their system/network/hardware. There's nothing wrong with that necessarily, but it is a pretty fundamental difference.

One concern I've had with building LLM features is whether my customers would be okay with me giving their data over to the LLM vendor. Say I'm building a tool for data analysis, is it really okay to a customer for me to give their table schemas or access to the data itself to OpenAI, for example?

I rarely hear that concern raised though. Similarly when I was doing consulting recently, I wouldn't use copilot on client projects as I didn't want copilot servers accessing code that I don't actually own the rights to. Maybe its over protective though, I have never heard anyone raise that concern so maybe its just me.

Re: We were wrong about GPUs

#375

Post by closewith »

PaulHoule wrote: post #173
sanderjd wrote: post #145

Earlier quoted context omitted.

Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!

This mainframe system from the 1990s was so much simpler https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parall... even if it wasn't as scalable as Kube. One the other hand, a cluster of 32 CMOS mainframe could handle any commercial computing job that people were doing in the 1990s.

Seems the causality is going the wrong direction there. Commercial jobs were limited by mainframe constraints, so that's where job sizes topped out.

Re: We were wrong about GPUs

#376

Post by closewith »

sgarland wrote: post #207
SkyPuncher wrote: post #199

Earlier quoted context omitted.

It all takes time, mental energy, etc. Different environments require different tradeoffs. The vast majority of startups will die before their systems engineering becomes a problem.

Constant firefighting because you engineered a pile of shit also takes time and mental energy.

Only if you let it. None of that is your problem - just consume the task queue.

Unless of course you're in a leadership role, in which case it's going to be priority #1,000 in 99.9% of cases.

Re: We were wrong about GPUs

#377

Post by bonzini »

transpute wrote: post #371
bonzini wrote: post #364

Earlier quoted context omitted.

NVIDIA support is not special as far as QEMU is concerned—the special parts are all in their proprietary device driver, and they talk to QEMU via the VFIO infrastructure for userspace drivers. They just reimplemented the same thing in Cloud Hypervisor. Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.

> NVIDIA.. just reimplemented the same thing in Cloud Hypervisor. Was that recent, with MIG support for GPGPU partitioning? Is there a public mailing list thread or patch series for that work? Nvidia has a 90-page deployment doc on vCS ("virtual compute server") for RedHat KVM, https://images.nvidia.com/content/Solutions/data-center/depl...

Not NVIDIA; fly.io reimplemented the parts that CH didn't already have. I know that CH is developed on GitHub but I don't know whether the changes are public or in-house.

That said, the slowness of QEMU's startup is always exaggerated. Whatever they did with CH they could have done with QEMU.

Re: We were wrong about GPUs

#378

Post by StefanBatory »

godelski wrote: post #293
ketzo wrote: post #282

Earlier quoted context omitted.

And if more systems engineers had more design knowledge, navigating the AWS console wouldn’t be like walking on hot coals. But it’s still $X0 billion/year business! We’re all different at good things, and it’s usually better to lean into your strengths than it is to paper over your weaknesses. We can wish everyone were good at everything, or we can try to actually get things done.

> We can wish everyone were good at everything, or we can try to actually get things done. False dichotomy. There's no reason we can't have both. I want to be clear, there's no perfect code or a perfect understanding or any of that. But the complaint here about not knowing /enough/ fundamentals is valid. There is some threshold which we should recognize as a minimum. The disagreement is about where this threshold is,…

> False dichotomy. There's no reason we can't have both.

I'd kinda want to argue with that - it is true, but we don't live in vacuum. Most programmers (me included, don't worry) aren't that skilled, and after work not everyone will want to study more. This is something that could be resolved by changing cultural focus, but like other things involving people, it's easier to change the system/procedures than habits.

Re: We were wrong about GPUs

#379

Post by yungporko »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

increasingly small is right. i'm definitely part of that former group but sadly more and more these days i just feel dumb for being this way. it usually just means that i'm less productive than my colleagues in practice as i'm spending time figuring out how things work while everybody else is pushing commits. maybe if we were put in a hypothetical locked room with no internet access i'd have a slightly easier time than them but that's not helpful to anybody.

once upon a time i could have said that it's better this way and that everybody will be thankful when i'm the only person who can fix something, but at this point that isn't really true when anybody can just get an LLM to walk them through it if they need to understand what's going on under the hood. really i'm just a nerd and i need to understand if i want to sleep at night lol.

Re: We were wrong about GPUs

#380

Post by mwcampbell »

post-it wrote: post #218
calvinmorrison wrote: post #210

Earlier quoted context omitted.

The difference between a good developer and a bad is understanding the stack. Not necessarily an expert but I spend a lot of time debugging for random issues and it could be dns or a file locking issue or a network or a api or parsing EDI whatever. Most recently I found a bug in software that had to do with how Windows runs 32 bit mode on 64 bit. I've never used windows professionally and I have only had unix machine…

It's great that you were able to debug that. It may have come at an opportunity cost of being able to solve some more specialized problem within your domain. In my job I develop a React Native app. I also need to have a decent understanding of iOS and Android native code. If I run into a bug related to how iOS runs 32 bit vs 64 bit software? Not my problem, we'll open a ticket with Apple and block the ticket in our s…

> we'll open a ticket with Apple and block the ticket in our system.

Wouldn't it be annoying to be blocked on Apple rather than shipping on your schedule?

Post reply on HN