Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

551–560 of 604 posts

Re: We were wrong about GPUs

#551

Post by anthk »

sroussey wrote: post #130
lifeisstillgood wrote: post #96

Earlier quoted context omitted.

Yes - but the amount of code that loves Amdahls law, is tiny compared to amount code churned out each day that can never run parallel over 1M cores - no matter how clever a compiler gets. I cannot work out if we pack enough parallel problems in the world or just lack a programming language to describe them

Computers are still Von-Neumann machines, and other architectures lost out due to the great returns on investment for that architecture. However, in the AI world, this might not be the case. For instance, neuromorphic computing is one example, and there are others. Or back to analog again! Superposition is instant—no slow adders with carry bits to propagate! Who knows. Fun times!

Not always. Modern computers are like several computers networked into one, if you think about it. Since the DMA, that approach is not valid. Today we have IOMMU's. The 9front/plan9 guys are trying to write a kernel (Nix) which wants to exploit every concurrent core of your CPU at crazy scaling levels.

Re: We were wrong about GPUs

#552

Post by techpression »

lovich wrote: post #537
remram wrote: post #505

Earlier quoted context omitted.

So you're comparing Kubernetes to what? Not running services at all? In that case I agree, you're going to have to set up Linux, find a storage solution, etc as part as your setup. Then write your app. It's a lot of work. But would I say that your entire Linux installation and the cloud it runs on is part of Kubernetes? No.

> So you're comparing Kubernetes to what? Not running services at all? Surprisingly there were hosted services on the internet prior to kubernetes existing. Hell, I even have reason to believe that the internet may possibly predate Docker

Next you’re going to claim the internet existed before Google too.

Re: We were wrong about GPUs

#553

Post by swiftcoder »

dzikimarian wrote: post #398
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

One may think Kubernetes is complex (I agree), but I haven't seen alternative that simultaneously allows to: * Host hundreds or thousands of interacting containers across multiple teams in sane manner * Let's you manage and understand how is it done in the full extent. Of course there are tons of organizations that can (and should) easily resign from one of these, but if you need both, there isn't better choice right…

> Host hundreds or thousands of interacting containers across multiple teams in sane manner

I mean, if that's your starting point, then complexity is absolutely a given. When folks complain about the complexity of Kubernetes, they are usually complaining about the complexity relative to a project that runs a frontend, a backend, and a postgres instance...

Re: We were wrong about GPUs

#554

Post by westurner »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

IaaS or PaaS?

Who owns and depreciates the logs, backups, GPUs, and the database(s)?

K8s docs > Scheduling GPUs: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus... :

> Once you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu.

> You can consume these GPUs from your containers by requesting the custom GPU resource, the same way you request cpu or memory

awesome-local-ai: Platforms / full solutions https://github.com/janhq/awesome-local-ai?platforms--full-so...

But what about TPUs (Tensor Processing Units) and QPUs (Quantum Processing Units)?

Quantum backends: https://github.com/tequilahub/tequila#quantum-backends

Kubernetes Device Plugin examples: https://kubernetes.io/docs/concepts/extend-kubernetes/comput...

Kubernetes Generic Device Plugin: https://github.com/squat/generic-device-plugin#kubernetes-ge...

K8s GPU Operator: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator...

Re: sunlight server and moonlight for 120 FPS 4K HDR access to GPU output over the Internet: https://github.com/kasmtech/KasmVNC/issues/305#issuecomment-... :

> Still hoping for SR-IOV in retail GPUs.

> Not sure about vCPU functionality in GPUs

Process isolation on vCPUs with or without SR-IOV is probably not as advanced as secure enclave approaches.

Intel SGX is a secure enclave capability, which is cancelled on everything but Xeon. FWIU there is no SGX for timeshared GPUs.

What executable loader reverifies the loaded executable in RAM after imit time ?

What LLM loader reverifies the in-RAM model? Can Merkle hashes reduce that cost; of nn state verification?

Can it be proven that a [chat AI] model hosted by someone else is what is claimed; that it's truly a response from "model abc v2025.02"?

PaaS or IaaS

Re: We were wrong about GPUs

#555

Post by chronid »

DrFalkyn wrote: post #526
amazingman wrote: post #487

Earlier quoted context omitted.

Controller in charge of a specific type of object watches a database table representing the object type. Database table represents the desired state of things. When entries to the table are CRUD-ed, that represents a change to the desired state of things. Controller interacts with the larger system to bring the state of things into alignment with the new desired state of things. "The larger system" is more controller…

Is there only a single controller ? What happens if goes down? If multiple controllers, how do they coordinate ?

The k/v store offers primitives to make that happen, but for non-critical controllers you don't want to deal with things like that they can go down and will be restarted (locally by kubelet/containerd) or rescheduled. Whatever resource they monitor will just not be touched until they get restarted.

Re: We were wrong about GPUs

#556

Post by chronid »

donutshop wrote: post #534
docandrew wrote: post #213

Earlier quoted context omitted.

“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…

... and kubernetes networking, service mesh, secrets management

You arent' forced to use service mesh and complex secrets management schemes. If you add them to the cluster is because you value what they offer you. It's the same thing as kubernetes itself - I'm not sure what people are complaining about, if you don't need what kubernetes offers, just don't use it.

Go back to good ol' corsync/pacemaker clusters with XML and custom scripts to migrate IPs and set up firewall rules (and if you have someone writing them for you, why don't you have people managing your k8s clusters?).

Or buy something from a cloud provider that "just works" and eventually go down in flames with their indian call centers doing their best but with limited access to engineering to understand why service X is misbehaving for you and trashing your customer's data. It's trade-offs all the way.

Re: We were wrong about GPUs

#557

Post by chronid »

swiftcoder wrote: post #553
dzikimarian wrote: post #398

Earlier quoted context omitted.

One may think Kubernetes is complex (I agree), but I haven't seen alternative that simultaneously allows to: * Host hundreds or thousands of interacting containers across multiple teams in sane manner * Let's you manage and understand how is it done in the full extent. Of course there are tons of organizations that can (and should) easily resign from one of these, but if you need both, there isn't better choice right…

> Host hundreds or thousands of interacting containers across multiple teams in sane manner I mean, if that's your starting point, then complexity is absolutely a given. When folks complain about the complexity of Kubernetes, they are usually complaining about the complexity relative to a project that runs a frontend, a backend, and a postgres instance...

In my last job we ran centralized clusters for all teams. They got X namespaces for their applications, and we made sure they could connect to the databases (handled by another team, though there were discussion of moving them onto dedicated clusters). We had basic configuration setup for them and offered "internal consultants" to help them onboard. We handled maintenance, upgrades and if needed migrations between clusters.

We did not have a cluster just for a single application (with some exceptions because those applications were incredibly massive in pod numbers) and/or had patterns that required custom handling and pre-emptive autoscaling (which we wrote code for!).

Why are so many companies running a cluster for each application? That's madness.

Re: We were wrong about GPUs

#558

Post by scosman »

ec109685 wrote: post #436
scosman wrote: post #368

> But inference latency just doesn’t seem to matter yet, so the market doesn’t care. This is a very strange statement to make. They are acting like inference today happens with freshly spun up VMs and model access over remote networks (and their local switching could save the day). It’s actually hitting clusters of hot machines with the model of choice already loaded into VRAM. In real deployments, latency can be sma…

I think what they mean about latency not mattering is that latency to the LLM provider doesn’t matter. So why run it yourself when there are API’s you can hit that provide a better overall experience (and seems to be dropping in cost 90% year over year).

Oh that makes more sense. My bad.

Re: We were wrong about GPUs

#559

Post by inkyoto »

anthk wrote: post #549
inkyoto wrote: post #525

Earlier quoted context omitted.

> […] self-directed learning. And this is whom I ended up hiring – young engineers with curious minds, who are willing to self-learn and are continuously engaged in the self-learning process. I also continuously suggest interesting, prospective, and relevant new things to take a look into, and they seem to be very happy to go away, pick the subject of study apart, and, if they find it useful, incorporate it into thei…

Maybe in the US. Any self-titled Engineer in Europe with no knowledge of CPU's, registers, stacks, concurrency, process management, scheduling, O-notation, dynamic systems, EE, and a bigass chunk of Math from Linear to Abstract it would be insta-bashed down in the spot with no degree at all. Here in Spain atthe most basic uni you are almost being able to write a Minix clone from scratch into some easy CPU (Risc-V may…

I am not based in the US, and I currently work for one of the top 100 universities of the world (the lower 50 part, though).

Re: We were wrong about GPUs

#560

Post by post-it »

mwcampbell wrote: post #380
post-it wrote: post #218

Earlier quoted context omitted.

It's great that you were able to debug that. It may have come at an opportunity cost of being able to solve some more specialized problem within your domain. In my job I develop a React Native app. I also need to have a decent understanding of iOS and Android native code. If I run into a bug related to how iOS runs 32 bit vs 64 bit software? Not my problem, we'll open a ticket with Apple and block the ticket in our s…

> we'll open a ticket with Apple and block the ticket in our system. Wouldn't it be annoying to be blocked on Apple rather than shipping on your schedule?

If we're blocked on Apple, so is everyone else. A key consideration in shipping high-level software is to avoid using niche features that the vendor might ignore if they're broken.
Post reply on HN