Out of curiosity, how much runway does fly.io have (without raising new funding?)
I'd guess significantly less after this debacle.
We were wrong about GPUs
91–100 of 604 posts
Re: We were wrong about GPUs
#92> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…
I used to think this, ut it only works if the abstractions hold - it’s like if we stopped random access memory and went back to tape drives suddenly abstractions matter.
My comment elsewhere goes into but more detail but basically silicon stopped being able to make single threaded code faster in about 2012 - we just have been getting “more parallel cores” since. And now at wafer scale we see 900,000 cores on a “chip”. When 100% parallel coding runs 1 million times faster than your competitors, when following one software engineering path leads to code that can run 1M X, then we will find ways to use that excess capacity - and the engineers who can do it get to win.
I’m not sure how LLMs face this problem.
Re: We were wrong about GPUs
#93> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
Aren’t we just continually moving up layers of abstractions? Most of the increasingly small group doesn’t concern itself with voltages, manually setting jumpers, hand-rolling assembly for performance-critical code, cache line alignment, raw disk sector manipulation, etc. I agree it’s worthwhile to understand things more deeply but developers slowly moving up layers of abstractions seems like it’s been a long term tre…
We certainly need abstractions for the first layer of the hardware. An abstraction of the abstraction can be useful if the first abstraction is very bad or very crude. But we are now at an abstraction of an abstraction x 8 or so. It's starting to get a bit over the top.
Re: We were wrong about GPUs
#94> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value.
Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering.
Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a week or two and cut our bill in half.
None of it really matters, though. We’re racing against competition and runway. A few days less runway isn’t going to break a startup. Not shipping as fast as reasonable will.
Re: We were wrong about GPUs
#95It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…
[deleted]
Re: We were wrong about GPUs
#96I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…
Plenty of non-IT applications use lots of cores, e.g. physics simulations, constraint solving, network simulation used to plan roads or electrical distribution, etc.
Yes - but the amount of code that loves Amdahls law, is tiny compared to amount code churned out each day that can never run parallel over 1M cores - no matter how clever a compiler gets.
I cannot work out if we pack enough parallel problems in the world or just lack a programming language to describe them
Re: We were wrong about GPUs
#97> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. This is baffling. What’s value proposition here? At some point customer will be directly asking an AI agent to create an app for them and it will take care of coding/deployment for them..
The value proposition is that you know how to fix something, when it eventually break, because you don’t fundamentally understand.
Re: We were wrong about GPUs
#98> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. (...) The other group (increasingly large) just wants to `git push` and be done with it
I think we're approaching the point where software development becomes a low-skilled job, because the automatic tools are good enough to serve business needs, while manual tools are too difficult to understand by anyone but a few chosen ones anyway.
Re: We were wrong about GPUs
#99Earlier quoted context omitted.
You described two points in an spectrum in which: One end is PaaS like Heroku, where you just git push. The other end is bare metal hosting. Every option you mentioned (VPS, Manages K8S, Self Hosted K8S, etc) they all fall somewhere between these two ends of the spectrum. If, a developer falls into any of these "groups" or has a preference/position on any of these solutions, they are just called juniors. Where you en…
This is news to us. Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not. You're not wrong that there's a Pa…
Ok, let me rephrase this:
> Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this?
to this:
Their audience has always been, and will always be application developers who prefer to do nothing except to build their main product.
> Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not.
I'm pretty much sure an application developer in this day and age has to know all of them, yes. Just like git.
Re: We were wrong about GPUs
#100Earlier quoted context omitted.
This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…
I used to think this, ut it only works if the abstractions hold - it’s like if we stopped random access memory and went back to tape drives suddenly abstractions matter. My comment elsewhere goes into but more detail but basically silicon stopped being able to make single threaded code faster in about 2012 - we just have been getting “more parallel cores” since. And now at wafer scale we see 900,000 cores on a “chip”…
That's why abstractions like PyTorch exist. You can write a few lines of Python and get good utilization of all those GPU cores.