Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

251–260 of 604 posts

Re: We were wrong about GPUs

#251

Post by epr »

pclmulqdq wrote: post #233
jimmaswell wrote: post #212

Earlier quoted context omitted.

If my understanding is right, the gist seems to be that you create one or more docker containers that your application can run on, describe the parameters they require e.g. ram size/cuda capability/when you need more instances, and kubernetes provisions them out to the machines available to it based on those parameters. It's abstract but very tractibly so IMO, and it seems like a sensible enough way to achieve load b…

It's worth noting that "container" and "process" are pretty similar abstractions. A lot of people don't realize this, but a container is sort of just a process with a different filesystem root (to oversimplify). That arguably is what a process should be on a server.

No, they are not. I'm not sure who started this whole container is just a process thing, but it's not a good analogy. Quite a lot of things you spin up containers for have multiple processes (databases, web servers, etc).

Containers are inherently difficult to sum up in a sentence. Perhaps the most reasonable comparison is to liken them to a "lightweight" vm, but the reasons people use them are so drastically different than vms at this point. The most common usecase for containers is having a decent toolchain for simple, somewhat reproducible software environments. Containers are mostly a hack to get around the mess we've made in software.

Re: We were wrong about GPUs

#252

Post by remram »

docandrew wrote: post #213
remram wrote: post #200

Earlier quoted context omitted.

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…

And system calls and filesystems and sockets and LVM and...

Sure at some point there are too many layers to count but I wouldn't say any of this is "Kubernetes". What people tend to be hung about is the difficulty of Kubernetes compared to `docker run` or `docker compose up`. That is what I am surprised about.

I never had any issue with kubelet, or kube-proxy, or CSI plugins, or CNI plugins. That is after years of running a multi-tenant cluster in a research institution. I think about those about as much as I think about ext4, runc, or GRUB.

Re: We were wrong about GPUs

#253

Post by jjmarr »

sroussey wrote: post #130
lifeisstillgood wrote: post #96

Earlier quoted context omitted.

Yes - but the amount of code that loves Amdahls law, is tiny compared to amount code churned out each day that can never run parallel over 1M cores - no matter how clever a compiler gets. I cannot work out if we pack enough parallel problems in the world or just lack a programming language to describe them

Computers are still Von-Neumann machines, and other architectures lost out due to the great returns on investment for that architecture. However, in the AI world, this might not be the case. For instance, neuromorphic computing is one example, and there are others. Or back to analog again! Superposition is instant—no slow adders with carry bits to propagate! Who knows. Fun times!

Most computers use modified Harvard architecture, funnily enough. There's a shared memory space like von Neumann, but separated caches for instructions and data.

It's the best of both worlds, because from the CPU's perspective it gets to have separate lanes for instructions and data, but from the programmer's perspective it's one memory.

Re: We were wrong about GPUs

#254

Post by cmdtab »

tptacek wrote: post #78
BoorishBears wrote: post #72

Earlier quoted context omitted.

As someone who deploys a lot of models on rented GPU hardware, their pricing is not realistic for continous usage. They're charging hyperscaler rates, and anyone willing to pay that much won't go with Fly. For serverless usage they're only mildly overpriced compared to say Runpod, but I don't think of serverless as anything more than an onramp to renting dedicated machine, so it's not surprising to hear it's not taki…

My thing here is just: people self-hosting LLMs think about performance in tokens/sec, and we think about performance in terms of ms/rtt; they're just completely different scales. We don't really have a comparative advantage for developers who are comfortable with multisecond response times. And that's fine!

That reminds me when cloudflare launched their workers gpu product, it was specifically aimed at running models and the pricing was abstracted and based on model output. Did you look what they were doing when building gpu machines?

https://blog.cloudflare.com/workers-ai/

Re: We were wrong about GPUs

#255

Post by Spooky23 »

mdaniel wrote: post #237
Spooky23 wrote: post #138

Earlier quoted context omitted.

I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong. I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GC…

Do you happen to have a link for your HP t6x0 reference? I tried https://www.ebay.com/sch/i.html?_nkw=hp+thin+client+32gb+-(4... and there seemed to be plenty with 32GB of storage but none that I could find with that much RAM

Sorry I was imprecise. I typically buy the cheapest one I can find with a power supply. Last year t630 were the sweet spot. They typically ship with 4 or 8… the windows models have higher spec. I picked up a couple of t730s too. I add third party memory and storage if needed.

You get a super capable, low power device in the price footprint of a raspberry pie.

Re: We were wrong about GPUs

#256

Post by dijksterhuis »

aqueueaqueue wrote: post #73

> developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. Is there not a market for the kind data science stuff where GPUs help but you are not using an LLM. Like statistical models on large amounts of data and so on. Maybe fly.io customer base isn't that sort of user. But I was pushing a previous company to get AWS GPUs because it would save us money vs CPU for the workload.

not from fly.io, but my experience is that most data scientists will just prefer to lump it with the tools they know (pandas / R) on CPUs, rather than delving into things like rapids https://rapids.ai -- even if it makes things faster/cheaper.

I might have had a bad sample set so far. But the "doing statistics" bit seems to be the interesting thing for them. the tooling doesn't really factor into solutions/plans that often. and learning something new because "engineer say it shinier" doesn't really seem to motivate them much :/

Re: We were wrong about GPUs

#257

Post by Spooky23 »

c0balt wrote: post #196
Spooky23 wrote: post #138

Earlier quoted context omitted.

I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong. I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GC…

A hyperscaler (or a cloud providers in general) does not only sell you compute in terms of a compute node but rather in compute as a service. There are some value adds, like e.g., AWS cloud services, but on a pure compute level you pay for elasticity and reliability. A comparison between a cloud provider and your homelab also needs to account for connectivity, which likely is in strong favor (latency/ reliability) of…

Absolutely — they add a ton of value. So did IBM… and companies migrated to NT solutions that were half baked because they were cheap.

When I can get the equivalent of a Mac Mini in a super cheap price point… you’re going to have opportunities to attack those stratospheric cloud margins.

Re: We were wrong about GPUs

#258

Post by digdugdirk »

morphle wrote: post #248
digdugdirk wrote: post #228

Earlier quoted context omitted.

I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?

> No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases…

Interesting! Fingers crossed someone who's looking for your skillset finds your post.

What is your process to turn Mac minis into a cluster? Is there any special hardware involved? And you can get 100x tok/s vs others on comparable hardware, what do you do differently - hardware, software, something else?

Re: We were wrong about GPUs

#259

Post by montecarl »

beebaween wrote: post #61
mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.

Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.

Re: We were wrong about GPUs

#260

Post by ehnto »

throwaway2037 wrote: post #225
Moru wrote: post #93

Earlier quoted context omitted.

We certainly need abstractions for the first layer of the hardware. An abstraction of the abstraction can be useful if the first abstraction is very bad or very crude. But we are now at an abstraction of an abstraction x 8 or so. It's starting to get a bit over the top.

I disgree with your sentiment. One thing that is constant in my experience as a computer programmer, there are always "old" computer programmers complaining that there are too many abstractions.

You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it.

I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. Most of what I see is moving sideways, not up the stack. Covering more breadth not height or depth, of abstractions.

Post reply on HN