241–250 of 604 posts
Earlier quoted context omitted.
Do you mean 128GB? Not aware of any variant of the Macbook Pro with that much RAM.
192GB is available for the M2 Mac Studio.
> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
I'm not entirely certain. Or perhaps we're all part of both groups. I want to understand LLMs. I want to understand my compiler, my gc, my type system, my distributed systems. On the other hand, I don't really care about K8s or anything else, as long as I have something that works. Just let me `git push` and focus on making great things elsewhere.
this feels right to me. application development and platform development are both software development tasks, and lots of software devs do both. i like working on platform-level stuff, and i like building applications. but i like there to be a good distinction between the two, and when i'm working on application-level stuff, i don't want to have to think about the platform.
services like fly.io do a good job of hiding all the platform level work and just giving you a place to deploy your application to, so when they start exposing tools like GPUs that are more about building platforms than building applications, it's messy.
Earlier quoted context omitted.
I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong. I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GC…
Do you happen to have a link for your HP t6x0 reference? I tried https://www.ebay.com/sch/i.html?_nkw=hp+thin+client+32gb+-(4... and there seemed to be plenty with 32GB of storage but none that I could find with that much RAM
Just took a quick look- appears t730 is of DDR3 era and may only have a single slot.
t740 definitely has two slots https://www8.hp.com/h20195/v2/GetPDF.aspx/c06393061.pdf
Earlier quoted context omitted.
> they're willing to spend a lot of (usually their employer's) money May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.
If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.
Kubernetes is something you can hire for. A couple of linux boxes running all your server code in the most efficient way possible might save you operational costs, but it resigns you to being the one who has to maintain it. I've learned this the hard way - moving things to ECS as we scale up has allowed me give away responsibility for things. I understand that it's more complex, but i don't have to teach people now.
Earlier quoted context omitted.
I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…
“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…
Earlier quoted context omitted.
I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…
Because the root comment is mostly but not quite right: there are indeed a large subset of developers that aren't interested in thinking about infrastructure, but there are many subcategories of those people, and many of them aren't fly.io customers. A large number of people who are in that category aren't happy to let someone else handle their infra. They're not interested in infra in the sense that they don't belie…
I assume everyone wants to be in control of their environment. But with so many ways to compose your infra that means a lot of different things for different people.
Earlier quoted context omitted.
I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…
“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…
step one: draw a circle
step two: import the rest of the owl
Earlier quoted context omitted.
What will you pay me for the benchmarks, for the professional knowledge and analysis? I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).
I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?
I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice.
>any interesting use cases to massive amounts of memory outside of training?
Dozens, hundreds. Almost anything you use databases, CPUs, GPUs or TPUs for. 90% of computing is done on the wrong hardware, not just datacenter hardware.
The interesting use case we discussed here on HN last week was running full DeepSeek-R1 LLms on 778 GB fast DRAM computers locally. I benchmarked getting hundreds of tokens per second on a cluster of M4 Mac minis or a cluster of M2 Mac Studio Ultras where others reported 0.015 or 6 tokens per second on single machines.
I just heard of a Brazilian man who build a 256 Mac Mini cluster at double the cost that I would. He leaves $600K value on the table because he won't reverse engineer the instruction set, rewrite his software or even call Apple to negotiated a low price.
HN votes me down for commenting that I, a supercomputer builder for 43 years, can build better cheaper faster low power supercomputers from Mac Mini's and FPGA's than from any Nvidia, AMD or Intel state of the art hardware, it even beats the fastest supercomputer of the moment or the Cerebras wafer engine V3 (on energy. coding cost and performance per watt per dollar).
I design and build wafer scale 2 million core reconfigurable supercomputers for $30K a piece that cost $150-$300 million to mass produce. That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
As a consulting job I do benchmarks or build your on-prem hardware or datacenter. This job consists mainly teaching the customer's programming staf how to program massively parallel software or convincing the CEO not to rent cloud hardware but buy on-prem hardware. OP at Fly.io should have hired me, then he wouldn't have needed to write his blog post.
I replied to your comment in hope of someone hiring me when they read this.
Earlier quoted context omitted.
As someone who deploys a lot of models on rented GPU hardware, their pricing is not realistic for continous usage. They're charging hyperscaler rates, and anyone willing to pay that much won't go with Fly. For serverless usage they're only mildly overpriced compared to say Runpod, but I don't think of serverless as anything more than an onramp to renting dedicated machine, so it's not surprising to hear it's not taki…
> GPU workloads tend to have terrible cold-start performance by their nature My Fly machine loads from turned off to first inference complete in about 35 seconds. If it’s already running, it’s 15 seconds to complete. I think that’s pretty decent.
And with the premium for per-second GPUs hovering around 2x that for hourly/monthly rentals, it gets even harder for products with scale to justify.
You'd want to have a lot of time where you're scaled to 0, but that in turn maps to a lot of cold starts.
Earlier quoted context omitted.
“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…
These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them b…