Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

481–490 of 604 posts

Re: We were wrong about GPUs

#481

Post by jrockway »

remram wrote: post #414
cuu508 wrote: post #299

Earlier quoted context omitted.

Hence > Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

CSI and CNI do about as much magic as `docker volume` and `docker network`. People act like their web framework and SQL connection pooler and stuff are so simple, while Kubernetes is complex and totally inscrutable for mortals, and I don't get it. It has a couple of moving parts, but it is probably simpler overall than SystemD.

> People act like their web framework and SQL connection pooler and stuff are so simple

I'm just sitting here wondering why we need 100 billion transistors to move a piece of tape left and right ;)

Re: We were wrong about GPUs

#482

Post by djhworld »

I get the impression that running LLMs is a pain in general, always seems to need the right incantation of nvidia drivers, linux kernel and a boat load of VRAM, along with making sure the Python ecosystem or whatever you are running for inference has the right set of libaries and then if you want multi-tenant processing across VMs - forget it or pay $$$ to nvidia.

The whole cloud computing world was built on hypervisors and CPU virtualization, I wonder if we'll see a similar set of innovations for GPUs at commodity level pricing. Maybe a completely different hardware platform will emerge to replace the GPU for these inference workloads. I remember reading about Google's TPU hardware and was thinking that would be the thing - but I've never seen anyone other than Google talk about it.

Re: We were wrong about GPUs

#483

Post by flockonus »

It feels like giving up on this a bit too soon? I mean, they realized the problem quite right.. their offering doesn't entirely makes sense for their audience when it comes to GPU.

_But_ the demand of open source models is just beginning. If they really have a big inventory of GPUs under-utilized and users want particular solutions on demand.... give it to them???

Like TTS STT video creation, real time illustration enhancement, deepseek and many others. You guys are great at devops, make useful offerings on demand, similar to what HuggingFace offers, no???

Re: We were wrong about GPUs

#484

Post by ukuina »

fmbb wrote: post #442
chucky_z wrote: post #440

Earlier quoted context omitted.

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a…

OK but the alternative if you think Kubernetes is too much magic when you want to operate hundreds of clusters with tens of thousands of nodes is? Some bash and Ansible and EC2? That is usually what Kubernetes haters suggest one does to simplify.

HashiCorp Nomad?

Re: We were wrong about GPUs

#485

Post by dartos »

dzikimarian wrote: post #459
dartos wrote: post #420

Earlier quoted context omitted.

I think the developers who care about knowing how their code works tend to not want hyperscale setups anyway. If they understood their system, odds are they’d realize that horizontal scaling with few, larger services is plenty scalable. At those large orgs, the individual developer doesn’t matter at all and the EMs will opt for faster release cycles and rely on internal platform teams to manage k8s and things like it…

Exact opposite - k8s allows developers to actually tailor containers/pods/deployments themselves, instead opening tickets to have it configured on VM by platform team. Of course there are simpler container runtimes, but they have issues with scale, cost, features or transparency of operation. Of course they can be good fit if you're willing to give up one or more of these.

> k8s allows developers to actually tailor containers/pods/deployments themselves

Yes, complex tools tend to be powerful.

But when I say “devs who care about knowing how their code works” I’m also referring to their tools.

K8s isn’t incomprehensible, but it is very complex, especially if you haven’t worked in devops before.

“Devs who care…” I would, assume, would opt for simpler tools.

I know I would.

Re: We were wrong about GPUs

#487

Post by amazingman »

abustamam wrote: post #455
formerly_proven wrote: post #443

Earlier quoted context omitted.

I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all. That being said, what people tend to build on top of that foundation is a somewhat different story.

I asked chatgpt the other day to explain to me Kubernetes. I still don't understand it. Can you share with me what clicked with you, or resources that helped you?

Controller in charge of a specific type of object watches a database table representing the object type. Database table represents the desired state of things. When entries to the table are CRUD-ed, that represents a change to the desired state of things. Controller interacts with the larger system to bring the state of things into alignment with the new desired state of things.

"The larger system" is more controllers in charge of other object types, doing the same kind of work for its object types

There is an API implemented for CRUD-ing each object type. The API specification (model) represents something important to developers, like a group of containers (Pod), a load balancer with VIP (Service), a network volume (PersistentVolume), and so on.

Hand wave hand wave, Lego-style infrastructure.

None of the above is exactly correct (e.g. the DB is actually a k/v store), but it should be conceptually correct.

Re: We were wrong about GPUs

#488

Post by volkk »

beebaween wrote: post #61
mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.

i use them a lot and constantly forget to turn mine off and it just drains my credits. i really need to write a job to turn them off when it's idle for longer than 20minutes

Re: We were wrong about GPUs

#489

Post by morphle »

sota_pop wrote: post #466
morphle wrote: post #263

Earlier quoted context omitted.

I discussed all the points you ask about in my HN postings last month, but never in enough detail so you must ask me to specify and that's when people hire me. As you can see from this comments thread, most people, especially programmers, lack the knowledge we computer scientist, parallel programmers and chip or hardware designers have. >What is your process Science. To measure is to know, my prof always said. To ans…

Sounds interesting, but I don’t see any HN submissions on your profile last month. Are you referring to comments you made?

>Are you referring to comments you made?

Yes. Several pages of comments about M4 clusters, wafer scale integrations and a few about DeepSeek.

https://news.ycombinator.com/threads?id=morphle (a few pages- press more).

https://news.ycombinator.com/item?id=42799072

Re: We were wrong about GPUs

#490

Post by tptacek »

coldpie wrote: post #175
tptacek wrote: post #36

Earlier quoted context omitted.

I'm a little sore the FOIA thing didn't make it up here, but Kurt's post did. ;)

Got a link? I flipped through the last dozen or so blog posts and none of them hit on a search for "FOIA".

This is all too painful for me to talk about.
Post reply on HN