Earlier quoted context omitted.
“I was wrong.” Closely followed by, “I was right.” :)
"Told you so."
That's personal pronouns.
I identify as a consipracy theorist, my pronouns are: Told/You/So.
We were wrong about GPUs
201–210 of 604 posts
Re: We were wrong about GPUs
#202Earlier quoted context omitted.
Nah, fly.io has a company culture that is all about having lots of bugs and issues, and that includes blog posts. The idea that a cloud compute provider can’t make GPU compute into an profitable business is pretty laughable.
I have to agree with this. Look at GPU utilization at AWS, Azure, .. they are running close to 100%. for our p5 quota I had to talk to our TAM team on AWS, while most of our quota requests are instant usually.
But the people buying GPU’s on AWS are not the same market as the ones on fly.io
The whole thing is sorta antithetical.
Re: We were wrong about GPUs
#203Earlier quoted context omitted.
> There's still plenty of hardware-specific APIs, you still debug assembly when something crashes, you still optimize databases for specific storage technologies and multimedia transcoders for specific CPU architectures... You might, maybe, but an increasing proportion of developers: - Don't have access to the assembly to debug it - Don't even know what storage tech their database is sitting on - Don't know or even c…
This makes sense to me. When I optimize, the most significant gains I find are algorithmic. Whether it's an extra call, a data structure that needs to be tweaked, or just utilizing a library that operates closer to silicon. I rarely need to go to assembly or even a lower level language to get acceptable performance. The only exception is occasionally getting into architecture specifics of a GPU. At this point, optimi…
> At this point, optimizing compilers are excellent
the only people that say this are people who don't work on compilers. ask anyone that actually does and they'll tell you most compiler are pretty mediocre (tend to miss a lot of optimization opportunities), some compilers are horrendous, and a few are good in a small domain (matmul).
Re: We were wrong about GPUs
#204> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. I considered using a Fly GPU instance for a project and went with Hetzner instead. Fly.io’s GPU offering was just way too expensive to use for inference.
Hetzner is more expensive by default though? It starts at $200/month. Which is fine if you are running for 720 hours every month, but you can run more cheaply on fly if it doesn’t get used more than 150ish hours in a month.
Re: We were wrong about GPUs
#205I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…
Parallelization of loading spinners?
Re: We were wrong about GPUs
#206you guys have all this juicy GPU and infrastructure. why not offer models as apis? i would pay to have apis for: sam2, florence, blip, flux 1.1, etc. whatever use case I would have reached Fly for on GPU, i can't justify _not_ using Replicate. maybe Fly can do better offer premium queues for that with their juicy infra? you're right! as a software dev I see dockerization and foisting these models as a burden, not a n…
Maybe you can have claude build it on top of fly.io? ;)
Re: We were wrong about GPUs
#207Earlier quoted context omitted.
Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.
It all takes time, mental energy, etc. Different environments require different tradeoffs. The vast majority of startups will die before their systems engineering becomes a problem.
Constant firefighting because you engineered a pile of shit also takes time and mental energy.
Re: We were wrong about GPUs
#208Earlier quoted context omitted.
Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…
I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…
Yeah, I'm sure there are tricky details as in anything but the core idea doesn't sound that complicated to me. I've been looking into it a bit after seeing this fun video a while ago where a DOS BBS is ran on kubernetes.
Re: We were wrong about GPUs
#209My heart stopped for a moment when reading the title. I'm glad they haven't decided to axe GPUs, because fly GPU machines are FANTASTIC! Extremely fast to start on-demand, reliable and although a little bit pricy but not unreasonably so considering the alternatives. And the DX is amazing! it's just like any other fly machine, no new set of commands to learn. Deploy, logs, metrics, everything just works out of the box…
I just looked at their pricing and they don't list any GPUs at all that I could find.
Re: We were wrong about GPUs
#210Earlier quoted context omitted.
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
Yeah, I think this really exemplifies the "everyone more specialized than me doesn't get the bigger picture, and everyone less specialized than me is wasting their time" trope. Developers who don't want to deal with the nitty gritty in one area are dealing with it in another area. Everyone has 24 hours in a day.
The difference between a good developer and a bad is understanding the stack. Not necessarily an expert but I spend a lot of time debugging for random issues and it could be dns or a file locking issue or a network or a api or parsing EDI whatever. Most recently I found a bug in software that had to do with how Windows runs 32 bit mode on 64 bit. I've never used windows professionally and I have only had unix machines since I got a free Ubuntu CD. Yet I figured it out in like 20 minutes exploring the differences between the paths when running in two scenarios. Idk maybe I'm a genius, I don't think so, but I was able to solve the problem because I know just barely enough about enough things to poke shit and break them or make them light up. Compare that to a dev on my team who needed help writing a series of command line prompts to do a simple bit of textual adjustments and pipe some data around.
I'm not a even good developer. But I know enough to chime in on calls and provide useful and generally 'Wizarding' knowledge. Like a detective with a good hunch.
But yeah just autocomplete everything lol