Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

331–340 of 604 posts

Re: We were wrong about GPUs

#331

Post by lifeisstillgood »

t0mas88 wrote: post #312
lifeisstillgood wrote: post #307

Earlier quoted context omitted.

Edit: it’s worth expanding on the data centre costs issues - if it’s fair to say we have stalled on “free”speed ups for standard software (ie C/Linux) - that is clock speeds more or less stopped and we get more 64 bit cores but each core is more or less no faster (hand wavy), then the amount of clients that can be packed into a data centre stays the same - you are renting out CPUs in a Docker VM - that’s basically on…

Web applications and APIs for mobile apps are embarrassingly parallel, but many modern web languages and frameworks went back to the stone age on parallelism. Ancient Java servlets back in the early 2000s were more suitable for performance on current gen hardware than modern NodeJS, Python etc...

It’s worth thinking about how to architect some of this in the new hardware.

To go extreme, wafer scale silicon - 900,000 8 bit cores, 120GB sRam. Hand wave on the 8bit for now and just think how to handle facebook or e-commerce. A static site is simple if it fits inside 120GB, but facebook is low write high read (Inassume) - but fetching from / writing to memory - making that parallel means rewriting a lot of things .. and e-commerce - suddenly ACID does not parallelise easily.

I mean this all seems doable with fairly fundamental changes to architecture memory concepts and … rdbms is challenging.

But I think we are way past the idea that a web framework author can just fix it - this is deeper - a new OS a new compiler and so on - but I could be wrong.

Re: We were wrong about GPUs

#332

Post by PeterStuer »

It also seems they got caught in the middle of the system integrator vs product company dilemma.

To me fly's offering reads like a system integrator"s solution. They assemble components produced mainly by 3rd parties into an offered solution. The business model of a system integrator thrives on doing the least innovation/custom work possible for providing the offering. You posotion yourself to take maximal advantage of investments and innovations driven by your 3rd party suppliers. You want to be squarely on their happy path.

Instead this artcle reads like fly, with good intention, was trying to divert their tech suppliers offer stream into niche edge cases outside of maistream support.

This can be a valid strategy for products very late into their maturity lifecycle where core innovation is stagnant, but for the current state of AI with extremely rapid innovation waves coarsing through the market, that strategy is doomed to fail.

Re: We were wrong about GPUs

#334

Post by pk-protect-ai »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Fly.io probably runs it on Kubernetes as well. It can be something in the middle, like RunPod. If you select 8 GPUs, you'll get a complete host for yourself. Though there is a lot of stuff lacking at RunPod too. But Fly.io... First of all, I've never heard about this one. Second, the variety of GPUs is lacking. There are only 3 types, and the L40S on Fly.io is 61.4% more expensive than on RunPod. So I would say it is about marketing, marketplace, long-term strategy, and pricing. But it seems at least they made themselves known to me (I bet there others which heard about them first time today too).

Re: We were wrong about GPUs

#335

Post by lifeisstillgood »

zahlman wrote: post #143
lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. In 2012, 22nm architectures were new ( https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchite... ). Now we have 3nm architectures ( https://en.wikipedia.org/wiki/3_nm_process ). In what sense have nand gates "not gotten smaller"? My computer wa…

I am still feeling my way through these ideas, but think perhaps of an alternative universe where instead of getting cleverer with instruction pipelining (guessing what the next CPU will ask for and using the silicon to work that out), hardware had just added more parallel cores - so it did not need to guess the next instructions, it just went faster because the instructions were in parallel because we magically solved software and developers.

You could have a laptop with 1000 cores on it - simple 32/64 bit CPUs that just ran full pelt.

The lack of parallelism drove decisions to take silicon and make it do stuff that was not run everything faster. But to focus on getting one instruction set through one core faster.

AI has arrived and found a world of silicon that it by coincidence can use every transistor for going full pelt - and the CPUs we think of in our laptops are using only a fraction of their transistors for full pedal to the metal processing and the rest is … legacy??

Re: We were wrong about GPUs

#336

Post by baq »

motorest wrote: post #315
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

> Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… I would ask a different question. How many people actually need to understand implementation details of Kubernetes? Look at any company. They pay engineers to maintain a web app/backend/mobile app. They want features to be rolled out, and they want their services to be up…

When they get paged three nights in a row and can’t figure out why.

Re: We were wrong about GPUs

#337

Post by zwnow »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

Also "They want LLMs" lol. I cant remember being asked. I dont want to use AI for coding.

Re: We were wrong about GPUs

#338

Post by 3836293648 »

Rury wrote: post #219
y1n0 wrote: post #174

Earlier quoted context omitted.

At the company I work for, I routinely mock the software devs for solving every problem by adding yet another layer of abstraction. The piles of abstractions these people levy is mind numbingly absurd. Half the things they are fixing, if not more, are created by the abstractions in the first place.

Yeah, I remember watching a video of (I think?) a European professor who helped with an issue devs were having in developing The Witness. Turns out they had a large algorithm they developed in high level code (~2000 lines of code? can't remember) to place flora in the game world, which took minutes to process, and it was hampering productivity. He looked at it all, and redid almost all of it in something like Frankly…

It's Casey Muratori, he's an American gamedev, not a professor. The video was a recorded guest lecture for a uni in the Netherlands though.

And it wasn't redone in assembly, it was C++ with SIMD intrinsics, which might as well just be assembly.

https://www.youtube.com/watch?v=Ge3aKEmZcqY&list=PLEMXAbCVnm...

Re: We were wrong about GPUs

#339

Post by baq »

mschuster91 wrote: post #310
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

> Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"... In the end it's a scheduler for Docker containers on a bunch of virtual or bare metal machines. Once you get that in your head life becomes much more easy. The only thing I'd really love to see from an ops perspective is a way to force-revive crashed containers for debugging. Yes,…

Yeah. In the whole cattle/pet discourse the fact that you need to take some cattle to the vet for diagnosis got lost. Very operator-centric thinking, I get where it’s coming from, but went a bit too far.

Re: We were wrong about GPUs

#340

Post by pdpi »

adrian_b wrote: post #314
Panzer04 wrote: post #121

Earlier quoted context omitted.

The unsung hero of early computing was Dennard scaling. Taking CPUS from 10MHz to 2GHz, all alongside massive per-clock efficiency improvements must have been a crazy time. From a 50MHz 486 in 1990 to a 1.4GHz P3 in 2000 is a factor of 28 improvement in speed solely due to clock speed! Add on all the other multiplicative improvements from IPC...

The greatest increase in clock frequency has been in the decade 1993-2003, when the clock frequency has increased 50 times (from a 66 MHz Pentium to a 3.2 GHz Pentium 4). Since then, in more than 20 years, the clock frequency has increased only 2 times, while in the previous decade (1983-1993) it had increased only about 5 times, where a doubling of the clock frequency (33 to 66 MHz) had occurred between 1989 and 199…

> while 80486 has reached 50 GHz only in 1991, not in 1990.

Your typo got me wondering — what would the performance of an actual 50GHz 486 look like compared to modern single-core performance?

The lack of speculative execution combined with atrocious memory latencies and next to no cache should be enough to annihilate most if not all of the advantage from the faster clock — CPU is just going to be idling waiting for data. Then there’s the amount of work you can get done per cycle, and SIMD, and…

Post reply on HN