Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

271–280 of 604 posts

Re: We were wrong about GPUs

#271

Post by fpoling »

daveguy wrote: post #141
fphhotchips wrote: post #119

Earlier quoted context omitted.

> There's still plenty of hardware-specific APIs, you still debug assembly when something crashes, you still optimize databases for specific storage technologies and multimedia transcoders for specific CPU architectures... You might, maybe, but an increasing proportion of developers: - Don't have access to the assembly to debug it - Don't even know what storage tech their database is sitting on - Don't know or even c…

This makes sense to me. When I optimize, the most significant gains I find are algorithmic. Whether it's an extra call, a data structure that needs to be tweaked, or just utilizing a library that operates closer to silicon. I rarely need to go to assembly or even a lower level language to get acceptable performance. The only exception is occasionally getting into architecture specifics of a GPU. At this point, optimi…

The need to expose SSE instruction to system languages tells that compilers are not good at translating straightforward code into optimal machine code. And using SSE properly allows often to speed up the code by several times.

Re: We were wrong about GPUs

#272

Post by [deleted] »

freedomben wrote: post #269
ffsm8 wrote: post #266

Earlier quoted context omitted.

But you just said that you had issues with ceph? How is that not a CSI problem? And CNI problems are extremely normal. Pretty much anyone that didn't just use weavenet and called it a day has had to spend quiet a bit of time to figure it out. If you already know networking by heart it's obviously going to be easier, but few devs do.

Very fair, although with managed services which are increasingly available, you don't typically need to think about CSI or CNI.

[deleted]

Re: We were wrong about GPUs

#273

Post by BoorishBears »

akoculu wrote: post #3

I spent a month setting up serverless endpoint for a custom model last year with Runpod. It was expensive and unreliable, in addition to long cold boot times. The product was unusable even as a prototype, to cover the costs, I'd have to raise money first. In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expens…

Fwiw Runpod also has a startup program.

Ironically GCP and AWS GPUs are so overpriced that getting even half the number of credits from Runpod is like a 4x increase in "GPU runway", especially with .44/hr A40s.

Re: We were wrong about GPUs

#274

Post by BoorishBears »

Aeolun wrote: post #188
johntash wrote: post #26

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(

That sounds kinda weird to me. I use fly.io exactly because the pricing works out for hobby use. I can enable my machine for a few hours, run a bunch of inference, and turn it off again. The whole auto start/stop thing makes it seamless too.

Vast has a datacenter H200 for less than what their A100 goes for.

Re: We were wrong about GPUs

#275

Post by BoorishBears »

Lyngbakr wrote: post #122
johntash wrote: post #26

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(

Who do you use instead for hobby projects?

Not them, but Runpod and Vast are my gotos. Runpod cost slightly more, but is in turn more reliable, so for "hobby pro" I'd go with them, otherwise Vast.

Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.

Re: We were wrong about GPUs

#276

Post by freedomben »

tptacek wrote: post #106
emilsedgh wrote: post #99

Earlier quoted context omitted.

Ok, let me rephrase this: > Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this? to this: Their audience has always been, and will always be application developers who prefer to do nothing except to build their main product. > Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Doc…

No. It is definitely not the case that the modal developer today needs to know Docker. If only! It's a huge pain point for us with the PaaS customer cohort.

Agreed. I have been amazed at how Little most developers seem to know docker now. It seemed like more people understood docker 5 to 10 years ago than they do now. I'm not sure why this regression, but it is definitely been my experience

Re: We were wrong about GPUs

#277

Post by steve_adams_86 »

chii wrote: post #270
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. tell me whether there's many brick layers who wants to understand the chemical composition of their bricks.

I’ve never laid bricks but in other trades I’ve worked in, well, a lot of people understood basics of the chemistry of the products we used. It’s useful to understand how they work together safely, if they can be exposed to different environments, if they’re heat-safe, cold-safe, do they off-gas, etc.

Paints, wood finishes, adhesives, oils, abrasives, you name it. You generally know at least a bit about what’s in it. I can’t say everyone I’ve worked with wanted to know, but it’s often intrinsic to what you’re doing and why. You don’t just pull a random product off a shelf and use it. You choose it, quite often, because of its chemical composition. I suspect it’s not always thought of this way, though.

This is the same with a lot of artistic mediums as well. Ceramicists often know a lot more than you’d expect about what’s in their clay and glazes. It’s really cool.

I’m not trying to be contrarian here. I know some people don’t care at all, and some people use products because it’s what they were told to do and they just go with it. But that wasn’t my experience most of the time. Maybe I got lucky, haha.

Re: We were wrong about GPUs

#278

Post by burnto »

I think they’re too early for their core market. It’s taking indie and 0-1 devs awhile to dig into ML because it’s a huge complex space. But some of us are starting to put together interesting little pipelines with real, solid applications.

Re: We were wrong about GPUs

#279

Post by fpoling »

c0balt wrote: post #196
Spooky23 wrote: post #138

Earlier quoted context omitted.

I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong. I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GC…

A hyperscaler (or a cloud providers in general) does not only sell you compute in terms of a compute node but rather in compute as a service. There are some value adds, like e.g., AWS cloud services, but on a pure compute level you pay for elasticity and reliability. A comparison between a cloud provider and your homelab also needs to account for connectivity, which likely is in strong favor (latency/ reliability) of…

Server hardware is reliable. For connectivity price-wise I think the sweet spot presently is to host own hardware in the data center and have a system administrator that lives not far away. I worked before for a company that was doing things like that while having millions of active users. It costed them at least 5 times less then it would be with a cloud provider. And then when they got a better deal with another data center the migration was not much more complex then moving server boxes in a van and changing ip addresses for load balancers.

Re: We were wrong about GPUs

#280

Post by anon373839 »

washadjeffmad wrote: post #118
zozbot234 wrote: post #63

Earlier quoted context omitted.

The best bang for the buck on VRAM is a maxed out Mac Studio.

No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.

I'm hoping to see some smaller MoE models released this year, trained with more recent recipes (higher quality data, much longer pretraining). Mixtral 8x7B was impressive when it came out, but the exact same architecture could be a lot more powerful today, and would run quite fast on Apple Silicon.
Post reply on HN