Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

141–150 of 604 posts

Re: We were wrong about GPUs

#141

Post by daveguy »

fphhotchips wrote: post #119
serviceberry wrote: post #101

Earlier quoted context omitted.

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

> There's still plenty of hardware-specific APIs, you still debug assembly when something crashes, you still optimize databases for specific storage technologies and multimedia transcoders for specific CPU architectures... You might, maybe, but an increasing proportion of developers: - Don't have access to the assembly to debug it - Don't even know what storage tech their database is sitting on - Don't know or even c…

This makes sense to me. When I optimize, the most significant gains I find are algorithmic. Whether it's an extra call, a data structure that needs to be tweaked, or just utilizing a library that operates closer to silicon. I rarely need to go to assembly or even a lower level language to get acceptable performance. The only exception is occasionally getting into architecture specifics of a GPU. At this point, optimizing compilers are excellent and probably have more architecture details baked into them than I will ever know. Thank you, compiler programmers!

Re: We were wrong about GPUs

#142

Post by nserrino »

cyberax wrote: post #40

Hah. We're doing AI, but we're doing vision-based stuff and not LLMs. For us, the problem has been deploying models. Google and AWS helpfully offered their managed LLM AI services, but they don't really have anything terribly more useful than just machines with GPUs. Which are expensive. I'm going to check fly.io...

What kind of models are you deploying and what type of problems are you having with deploying them?

Re: We were wrong about GPUs

#143

Post by zahlman »

lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there.

In 2012, 22nm architectures were new (https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchite...). Now we have 3nm architectures (https://en.wikipedia.org/wiki/3_nm_process). In what sense have nand gates "not gotten smaller"?

My computer was initially built in 2014 and the CPU runs up to 3 GHz (and I don't think it was particularly new on the market). CPUs made today overclock to 5+ GHz. In what sense did "single threaded execution more or less stop at 2 GHz and remain there"?

We might not be seeing exponential increases in e.g. transistor count with the same doubling period as before, but there has demonstrably been considerable improvement to traditional CPUs in the last 12+ years.

Re: We were wrong about GPUs

#144

Post by busterarm »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

It's very easy to understand once you invest a little bit of time.

That's assuming you have a solid foundation in the nuts and bolts of how computers work to begin with.

If you just jumped into software development without that background, well, you're going to end up in the latter pool of developers as described by the parent comment.

Re: We were wrong about GPUs

#145

Post by sanderjd »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!

Re: We were wrong about GPUs

#146

Post by busterarm »

Uvix wrote: post #116
_dark_matter_ wrote: post #74

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

The details matter because someone has to understand the details, and it's quicker and more cost-effective if it's the developer. At my job, a decade ago our developers understood how things worked, what was running on each server, where to look if there were problems, etc. Now the developers just put magic incantations given to them by the "DevOps team" into their config files. Most of them don't understand where th…

> (I'm part of said support team, but I really wish we didn't exist. We started to take over Ops responsibilities from a different team, but we ended up taking on Dev ones too and we never should've done that.)

There will always be work for people like us. It's not so bad. We're not totally immune to layoffs but for us they come several rounds in.

Re: We were wrong about GPUs

#147

Post by teaearlgraycold »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

It depends on the product you're building. At my last job we hosted bespoke controlnet-guided diffusion models. That means k8s+GPUs was a necessity. But I would have loved to use something simpler than k8s.

Re: We were wrong about GPUs

#148

Post by devmor »

The emotion in this article hits home to me. There have been several points in my career where I worked hard for long hours to develop a strong, clever solution to a problem that ultimately was solved by something cheap because the ultimate consumer didn’t care about what we expected them to care about.

It sucks from a business perspective of course, but it also sucks from the perspective of someone who takes pride in their work! I like to call it “artisan’s regret”.

Re: We were wrong about GPUs

#149

Post by jonathanlei »

thundergolfer wrote: post #65
onli wrote: post #22

Not sure about this: > like with our portfolio of IPv4 addresses, I’m even more comfortable making bets backed by tradable assets with durable value. Is that referencing the gpus, the hardware? If yes, why should they have a durable value? Historically hardware like that deprecated fast and reaches a value of 0, energy efficiency alone kills e.g. old server hardware. Something different here?

I think it's referencing only the IPv4 block, but it is a bit confusing. It doesn't make sense to be ref'ing the GPUs because their value is definitely not durable.

Absolutely - GPUs are definitely not a very liquid asset. As someone who works at a GPU neocloud provider (Voltage Park), server assets at scale definitely face a huge slippage, you can buy for $1 and get quotes for $1.50 but only be able to sell for $0.60

Re: We were wrong about GPUs

#150

Post by [deleted] »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

[deleted]
Post reply on HN