Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

591–600 of 604 posts

Re: We were wrong about GPUs

#591

Post by sethammons »

moondev wrote: post #387
sethammons wrote: post #362

Earlier quoted context omitted.

> Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs Uh, any time I run a distributed system and logs could appear on n nodes I need a log aggregator or I am tailing in n terminals. I almost only use Splunk. I tail logs in dev. Prod needs an aggregator. This has been my experience at 4 of my last 6 companies. The shit companies who had all the issues? L…

kubectl logs deployment my-multinode-deployment

That's a (crappy) aggregator technically.

Re: We were wrong about GPUs

#592

Post by akoculu »

BoorishBears wrote: post #273
akoculu wrote: post #3

I spent a month setting up serverless endpoint for a custom model last year with Runpod. It was expensive and unreliable, in addition to long cold boot times. The product was unusable even as a prototype, to cover the costs, I'd have to raise money first. In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expens…

Fwiw Runpod also has a startup program. Ironically GCP and AWS GPUs are so overpriced that getting even half the number of credits from Runpod is like a 4x increase in "GPU runway", especially with .44/hr A40s.

Yeah but the quality on Runpod is not reliable enough for productionizing it. Do you know a product that works reliably and it's built with Runpod serverless?

Re: We were wrong about GPUs

#594

Post by johntash »

mrkurt wrote: post #569
johntash wrote: post #325

Earlier quoted context omitted.

Hmm, having a machine for every process is part of it but I actually like that kind of isolation. Storage and bandwidth also add up fast. For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo. For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops. I recently tried experimenting with two different pro…

No problem at all! I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches".…

Tbh I haven't looked at Tigris at all. I still have my attic server deployed (just disabled/not in use) so I might give it a shot just to compare pricing. I do remember a decent portion of the cost being storage-related, so it's a good idea.

I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance.

> It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?

Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic.

> We are suboptimal for things that make more sense as a bunch of containers on one host.

I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.

Re: We were wrong about GPUs

#595

Post by garfieldnate »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

It's been a while since I tried, but my experience trying to manually set up GPUs was atrocious, and with investigation generally ending at the closed-source NVidia drivers it's easy to feel disempowered pretty quickly. I think my biggest learning from trying to do DL on a manually set up computer was simply that GPU setup was awful and I never wanted to deal with it. It's not that I don't want to understand it, but with NVidia software you're essentially not allowed to understand it. If open source drivers or open GPU hardware were released, I would gladly learn how that works.

Re: We were wrong about GPUs

#596

Post by alanmoraes »

aeyes wrote: post #262
layoric wrote: post #111

Earlier quoted context omitted.

I’ve been in similar situations, but details matter. If your scale up button is heavily abstracted services, your choice starts to become very different as the cost of reimplementing what the service does might be high enough that you end up with a no win situation of your own making. The closer your “Scale up” button is referencing actual hardware, the less of a problem it is.

That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door. Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data. When you get to that…

Why don't refactor just the parts the need it instead of rewriting everything?

Re: We were wrong about GPUs

#597

Post by gopher_space »

montecarl wrote: post #290
gopher_space wrote: post #281

Earlier quoted context omitted.

What's your workload and timeline? I'm wondering how much of that workload could be handled in-house.

Just currently exploring how custom AI workflows (e.g. text to sql, custom report generation using private data) can help given the current SOTA. Looking to develop tooling over the next 3-6 months. I'd like to see what we can come up with before dropping $50-100k on hardware.

I threw together a toy project to see if it would help me understand the basic concepts and my takeaway was that, if you can shape your input into something a dedicated classification model (e.g. YOLO for document layout analysis) can work with, you can farm each class out to the most appropriate model.

It turns out that I can run most of the appropriate models on my ancient laptop if I don't mind waiting for the complicated ones to finish. If I do mind, I can just send that part to OpenAI or similar. If your workflow can scale horizontally like my OCR pipeline crap, every box in your shop with RAM >= 16GB might be useful.

Apologies if this is all stuff you're familiar with.

Re: We were wrong about GPUs

#598

Post by aeyes »

alanmoraes wrote: post #596
aeyes wrote: post #262

Earlier quoted context omitted.

That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door. Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data. When you get to that…

Why don't refactor just the parts the need it instead of rewriting everything?

You can do small refactors here and there but usually things get complex after some time. My first advice was to not overthink on architecture in the beginning so it is inevitable that you will end up with something quite unorganized after a while. The assumption here is that in the beginning you won't know what the architecture will look like in the future because you are in startup/discovery mode.

Refactoring such a codebase while keeping everything running can be a monumental effort. I found it very hard to keep people who work on such a project motivated. Analyzing the use cases, coming up with a new design incorporating your learnings and then seeing clear progress towards the goal of a cleaner codebase is much more motivating. Engineers get the chance to do something new instead of "moving code around".

I'm not saying rewrite everything. When you get to this point it usually makes sense to start thinking about these abstractions which I advised to avoid in the beginning. You can begin to separate parts of the system by responsibility and then you rewrite just one part and give it a new API which other parts of your system will consume. Usually by that time you'll also want to restructure your database.

Re: We were wrong about GPUs

#599

Post by zeroq »

chasd00 wrote: post #467
zeroq wrote: post #308

Earlier quoted context omitted.

Having a team that runs the kubernetes for you and being on receiving end is indeed super easy. Need another microservice? Just add another repository, add short yaml, push it to CI and bam!, it's online. But setting it up is not a trivial task and often a recipe for disaster. I've seen a fair share of startups who took too much kool aid and wanted parrot FANG stacks just to discover they are burning tons of money ju…

The irony is the whole devops and cloud sales pitch was developers can do all this themselves and you no longer need an sysadmin team. Turns out you still do, it’s just called the devops/cloud team and not sys admin team.

just yesterday a friend messaged me that their cutting edge cloud systems based on kubernetes and latests achievements in microservice architecture will have... 2 days maintenance downtime.

Re: We were wrong about GPUs

#600

Post by fennecfoxy »

fiberhood wrote: post #286
mdaniel wrote: post #241

Earlier quoted context omitted.

I was curious "how bad is it?" and it seems $5500-ish https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15

$6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.

$6500!! You may as well buy 5x3090s for $1000 each for 120GB ram, spend the extra $1500 on the sundries.

Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened.

Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here?

That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible.

I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.

Post reply on HN