What a good and open and honest blog post. And I liked a lot the way it is interlinked with other interesting posts from that blog. I wish I'll have some time to read more articles from that blog.
We were wrong about GPUs
161–170 of 604 posts
Re: We were wrong about GPUs
#162If low cost GPUs are not what they are offering then what are they offering anymore that I wouldn't get a big cloud vender. This looks like self inflicted mortal wound.
Re: We were wrong about GPUs
#163Earlier quoted context omitted.
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!
It gives you control via abstractions. That’s fine, and I like K8s personally, but if you don’t understand the underlying fundamentals that it’s controlling, you don’t understand what it’s doing.
Re: We were wrong about GPUs
#164If low cost GPUs are not what they are offering then what are they offering anymore that I wouldn't get a big cloud vender. This looks like self inflicted mortal wound.
"Mortal wound" lol.
Re: We were wrong about GPUs
#165Re: We were wrong about GPUs
#166Re: We were wrong about GPUs
#167Nvidia deliberately makes this hard. Opportunity for Intel and AMD.
My takeaway from the article is that there's no real market for GPU VMs so it's pointless for Intel and AMD to make them work.
Re: We were wrong about GPUs
#168Earlier quoted context omitted.
> There's still plenty of hardware-specific APIs, you still debug assembly when something crashes, you still optimize databases for specific storage technologies and multimedia transcoders for specific CPU architectures... You might, maybe, but an increasing proportion of developers: - Don't have access to the assembly to debug it - Don't even know what storage tech their database is sitting on - Don't know or even c…
This makes sense to me. When I optimize, the most significant gains I find are algorithmic. Whether it's an extra call, a data structure that needs to be tweaked, or just utilizing a library that operates closer to silicon. I rarely need to go to assembly or even a lower level language to get acceptable performance. The only exception is occasionally getting into architecture specifics of a GPU. At this point, optimi…
> At this point, optimizing compilers are excellent and probably have more architecture details baked into them than I will ever know.
While modern compilers are great, you’d be surprised about the seemingly obvious optimizations compilers can’t do because of language semantics or the code transformations would be infeasible to detect.
I type versions of functions into godbolt all the time and it’s very interesting to see what code is/isn’t equivalent after O3 passes
Re: We were wrong about GPUs
#169Earlier quoted context omitted.
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!
IMO, it's rather hard to fully know all of kubernetes and what it's doing, and the kind of person who demands elegance in solutions will hate it.
Re: We were wrong about GPUs
#170I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…
[deleted]