We wrote all sorts of stuff this week and this is what gets to the front page. :P
Empathy. Such a seemingly good idea. Maybe just a little ahead of its time.
We were wrong about GPUs
261–270 of 604 posts
Re: We were wrong about GPUs
#262Earlier quoted context omitted.
I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a…
I’ve been in similar situations, but details matter. If your scale up button is heavily abstracted services, your choice starts to become very different as the cost of reimplementing what the service does might be high enough that you end up with a no win situation of your own making. The closer your “Scale up” button is referencing actual hardware, the less of a problem it is.
That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door.
Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data.
When you get to that point I have another pro tip: Don't refactor, just rewrite it and put all your learnings into the v2.
Re: We were wrong about GPUs
#263Earlier quoted context omitted.
> No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases…
Interesting! Fingers crossed someone who's looking for your skillset finds your post. What is your process to turn Mac minis into a cluster? Is there any special hardware involved? And you can get 100x tok/s vs others on comparable hardware, what do you do differently - hardware, software, something else?
I discussed all the points you ask about in my HN postings last month, but never in enough detail so you must ask me to specify and that's when people hire me.
As you can see from this comments thread, most people, especially programmers, lack the knowledge we computer scientist, parallel programmers and chip or hardware designers have.
>What is your process
Science. To measure is to know, my prof always said.
To answer your questions in detail, email me.
You first need to be specific. The problem is not how to turn Mac minis into a cluster, with or without custom hardware ( I do both) on code X or Y. Or how to optimize software or rewrite it from scratch (which its often cheaper).
First find the problem. In this case the problem is find the lowest OPEX and Capex to do the stated compute load versus changing the compute load. Turns out in a simulation or a cruder spreadsheet calculation it becomes clear that the energy cost dominates of hardware choice, it trumps the cost of programming, the cost of off the shelf hardware and the difference if you add custom hardware. M4's are lower power, lower OPEX and lower CAPEX especially if you rewrite your (Nvidia GPU) software. The problem is the ignorance of the managers and their employee programmers.
You can repurpose the 2 x 10 Gbps USB-C, the 10 Gbps Ethernet and the three 32 Gbps PCIe ports or Thunderbolts but you have to use better drivers. You need to weigh if double the 960 Gbps 16 GB unified memory for 2 x $400 is faster than 2 Tbps memory at 1.23 times the cost versus 3 x 4 x 32 Gbps PCIe 4.0 versus 3 x 120 Gbps unidirectionally is better for this particular algorithm and wheat changes if you uses both the 10 CPU cores, 10 x 400 GPU corses and 16 Neural Engine cores (at 38 trillion 16 bit OPS) will work batter than just the CUDA cores. Ususally the answers is: rewrite the alogoritm and use an adaptive compiler and then a cluster of smaller 'sweet spot' off the shelf hardware will outperform the most fancy high end hardware if the network is balanced. This varies at runtime so you'll only know if you now how to code. As Akan Kay said and Steve Jobs quoted: if your serious about software you should do your own hardware. If you can't, then you can approach the hardware with commodity components if that turns out to be cheaper. I estimate for $42K labour I can save you a few hundred $k.
Re: We were wrong about GPUs
#264Earlier quoted context omitted.
I’ve been in similar situations, but details matter. If your scale up button is heavily abstracted services, your choice starts to become very different as the cost of reimplementing what the service does might be high enough that you end up with a no win situation of your own making. The closer your “Scale up” button is referencing actual hardware, the less of a problem it is.
That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door. Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data. When you get to that…
This is exactly, precisely what my experience has been.
Re: We were wrong about GPUs
#265Earlier quoted context omitted.
This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…
> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…
These days even if one writes in machine code it will be quite far away from the real silicon as that code has little to do with what CPU is actually doing. I suspect that C source code from, say, nineties was closer to the truth than the modern machine code.
Re: We were wrong about GPUs
#266Earlier quoted context omitted.
“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…
And system calls and filesystems and sockets and LVM and... Sure at some point there are too many layers to count but I wouldn't say any of this is "Kubernetes". What people tend to be hung about is the difficulty of Kubernetes compared to `docker run` or `docker compose up`. That is what I am surprised about. I never had any issue with kubelet, or kube-proxy, or CSI plugins, or CNI plugins. That is after years of ru…
But you just said that you had issues with ceph? How is that not a CSI problem?
And CNI problems are extremely normal. Pretty much anyone that didn't just use weavenet and called it a day has had to spend quiet a bit of time to figure it out. If you already know networking by heart it's obviously going to be easier, but few devs do.
Re: We were wrong about GPUs
#267Earlier quoted context omitted.
This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.
Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.
I read through the FAQ and the answer is "no", but they say it basically as "nobody really cares what your data is".
I wouldn't put anything confidential through it.
Re: We were wrong about GPUs
#268> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
>who don't like "magic" and want to understand where their code is running and what it's doing.
I just made this point in a post on my substack. Especially in regulated industries, you NEED to the able to explain your AI to the regulator. You can't have a situation where a human say "Well, gee I don't know. The AI told me to do it."
Re: We were wrong about GPUs
#269Earlier quoted context omitted.
And system calls and filesystems and sockets and LVM and... Sure at some point there are too many layers to count but I wouldn't say any of this is "Kubernetes". What people tend to be hung about is the difficulty of Kubernetes compared to `docker run` or `docker compose up`. That is what I am surprised about. I never had any issue with kubelet, or kube-proxy, or CSI plugins, or CNI plugins. That is after years of ru…
But you just said that you had issues with ceph? How is that not a CSI problem? And CNI problems are extremely normal. Pretty much anyone that didn't just use weavenet and called it a day has had to spend quiet a bit of time to figure it out. If you already know networking by heart it's obviously going to be easier, but few devs do.
Very fair, although with managed services which are increasingly available, you don't typically need to think about CSI or CNI.
Re: We were wrong about GPUs
#270> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using.
tell me whether there's many brick layers who wants to understand the chemical composition of their bricks.