Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

531–540 of 604 posts

Re: We were wrong about GPUs

#531

Post by godelski »

StefanBatory wrote: post #378
godelski wrote: post #293

Earlier quoted context omitted.

> We can wish everyone were good at everything, or we can try to actually get things done. False dichotomy. There's no reason we can't have both. I want to be clear, there's no perfect code or a perfect understanding or any of that. But the complaint here about not knowing /enough/ fundamentals is valid. There is some threshold which we should recognize as a minimum. The disagreement is about where this threshold is,…

> False dichotomy. There's no reason we can't have both. I'd kinda want to argue with that - it is true, but we don't live in vacuum. Most programmers (me included, don't worry) aren't that skilled, and after work not everyone will want to study more. This is something that could be resolved by changing cultural focus, but like other things involving people, it's easier to change the system/procedures than habits.

Are you wanting to argue or discuss? You can agree in part and disagree with another part. Doesn't have to be an argument.

To your point I agree. I would argue that employers should be giving time for employees to better themselves. It's the nature of any job like this where innovation takes place. It's common among engineers, physicists, chemists, biologists, lawyers, pilots, and others to have time to learn. Doctors seem to be in the same boat as us and it has obviously negative consequences. The job requires continuous learning. And you're right, that learning is work. So guess who's supposed to pay for work?

Re: We were wrong about GPUs

#532

Post by seanp2k2 »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

Having worked with many of the latter and having had the displeasure of educating them on nix systems fundamentals: ugh, oof, I hate this timeline, yet I also feel a sense of job security.

We used to joke about this a lot when Java devs would have memory issues and not know how to adjust the heap size in init scripts. So many “CS majors” who are completely oblivious to anything happening outside of the JVM, and plenty happening within it.

Re: We were wrong about GPUs

#533

Post by ksec »

mwcampbell wrote: post #515
mike_hearn wrote: post #405

Earlier quoted context omitted.

Apple's chips are dramatically faster than any other kind. If you are single thread perf constrained and have the money, running workloads on Apple silicon can actually make sense.

> Apple's chips are dramatically faster than any other kind. Any idea why? Is it because of some patent they hold?

Nothing to do with that, they just have a head start in the right direction along with enough money to fund many iterations before it landed on Desktop or become accepted by 95% of people.

Qualcomm's current Snapdragon Elite Oryon 2 on Mobile, and ARM Cortex X925 or previously known as X5 are already close to Apple A17 level performance. So this is no longer something unique to Apple.

I just wish both design are more widely available. And for x86, Intel and AMD still haven't quite caught up. At least not in the next 2 years.

Re: We were wrong about GPUs

#534

Post by donutshop »

docandrew wrote: post #213
remram wrote: post #200

Earlier quoted context omitted.

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…

... and kubernetes networking, service mesh, secrets management

Re: We were wrong about GPUs

#535

Post by jchw »

motorest wrote: post #520
jchw wrote: post #392

Earlier quoted context omitted.

I'm not sure what your point is.

> I'm not sure what your point is. My point is that the mere notion of "a system that's actually big or complex enough to warrant using Kubernetes" is completely absurd, and communicates a high degree of complete cluelessness over the whole topic. Do you know what's a system big enough for Kubernetes? It's a single instance of a single container. That's it. Kubernetes is a container orchestration system. You tell it…

First of all, I don't really get the unnecessary condencension. I am not a beginner when it comes to Kubernetes and don't struggle to understand the concept at all. I first used Kubernetes at version 1.3 back in 2016, ran production workloads on it, contributed upstream to Kubernetes itself, and at one point even did a short bit of consulting for it. I am not trying to claim to be any kind of authority on Kubernetes or job scheduling as a topic, but when you talk down to people the way that you are doing to me, it doesn't make your point any better, it just makes you look like an insecure dick. I really tried to avoid escalating this on the last reply, but it has to be said.

Second of all, I don't really understand why you think I'd be blown away by the notion that you can use Kubernetes to run a single container. You can also open a can with a nuclear warhead, does not mean it makes any sense.

In production systems, Kubernetes and its ecosystem are very useful for providing the kinds of things that are table stakes, like zero-downtime deployments, metric collection and monitoring, resource provisioning, load balancing, distributed CRON, etc. which absolutely doesn't come for free either in terms of complexity or resource utilization.

But if all you need to do is run one container on a Raspberry Pi and don't care about any of that stuff, then even something stripped down like k3s is simply not necessary. You can use it if you want to, but it's overkill, and you'll be spending memory and CPU cycles on shit you are basically not using. Literally anything can schedule a single pod on a single node. A systemd Podman unit will certainly work, for example, and it will involve significantly less YAML as a bonus.

I don't think the point I'm making is particularly nuanced here. It's basically YAGNI but for infrastructure.

Re: We were wrong about GPUs

#536

Post by lovich »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing.

That problem started so long ago and has gotten so bad that I would be hard pressed to believe there is anyone on the planet who could take a modern consumer pc and explain what exactly is going on the machine without relying on any abstractions to understand the actual physical process.

Given that, it’s only a matter of personal preference on where you draw the line for magic. As other commenters have pointed out, your line allowing for Kubernetes is already surprising to a lot of people

Re: We were wrong about GPUs

#537

Post by lovich »

remram wrote: post #505
ffsm8 wrote: post #503

Earlier quoted context omitted.

In the context of this discussion, which is about the complexity of the k8s stack: yes. Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex. You could argue that multinode Paa…

So you're comparing Kubernetes to what? Not running services at all? In that case I agree, you're going to have to set up Linux, find a storage solution, etc as part as your setup. Then write your app. It's a lot of work. But would I say that your entire Linux installation and the cloud it runs on is part of Kubernetes? No.

> So you're comparing Kubernetes to what? Not running services at all?

Surprisingly there were hosted services on the internet prior to kubernetes existing. Hell, I even have reason to believe that the internet may possibly predate Docker

Re: We were wrong about GPUs

#538

Post by ianmcnaney »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

Somebody who doesn’t want to understand DNS, Linux, or anything beyond their framework is a hazard. They’re not able to do a competent code review on the vomit that LLMs produce. (Am I biased much?)

Re: We were wrong about GPUs

#540

Post by mike_hearn »

mwcampbell wrote: post #515
mike_hearn wrote: post #405

Earlier quoted context omitted.

Apple's chips are dramatically faster than any other kind. If you are single thread perf constrained and have the money, running workloads on Apple silicon can actually make sense.

> Apple's chips are dramatically faster than any other kind. Any idea why? Is it because of some patent they hold?

Combination of using the latest TSMC processes, a very wide design, very deep speculation pipelines, a weaker memory model than Intel, and lots of clever tricks of the usual sort for fast cpus, also a very high memory bandwidth.

I don't think it's anything to do with patents although I'm sure they have plenty.

Post reply on HN