Live data from Hacker News

We were wrong about GPUs

fly.io

571–580 of 604 posts

Re: We were wrong about GPUs

#571

Earlier quoted context omitted.

These days even if one writes in machine code it will be quite far away from the real silicon as that code has little to do with what CPU is actually doing. I suspect that C source code from, say, nineties was closer to the truth than the modern machine code.

Could you elaborate? I may very well just be ignorant on the topic. I understand that if you write machine code and run it in your operating system, your operating system actually handles its execution (at least, I _think_ I understand that), but in what way does it have little to do with what the CPU is doing? For instance, couldn't you still run that same code on bare metal? Again, sorry if I'm misunderstanding som…

The quest for performance has turned modern CPUs into something that looks a lot more like a JITed bytecode interpreter rather than the straightforward “this opcode activates this bit of the hardware” model they once were. Things like branch prediction, speculative execution, out-of-order execution, hyperthreading, register renaming, L1/2/3 caches, µOp caches, TLB caches... all mean that even looking at the raw machine code tells you relatively little about what parts of the hardware the code will activate or how it will perform in practice.

Re: We were wrong about GPUs

#572
post #200

Earlier quoted context omitted.

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a…

The wheels fall off kubernetes at around 10k nodes. One of the main limitations is etcd from my experience, google recently fixed this problem by making spanner offer an etcd compatible API: https://cloud.google.com/blog/products/containers-kubernetes...

Etcd is truly a horrible data store, even the creator thinks so.

Re: We were wrong about GPUs

#573

Earlier quoted context omitted.

We only run out of abstraction once there is stagnation and time to really bake. As long as some new thing is being invented in our industry, a new abstraction will be needed because the old one just can’t quite flex enough while being backwards compatible.

One problem is that we're building abstractions on top of older abstractions when that isn't most efficient. Suppose we write an HTTP server that can call CGI programs. Then we implement PHP as a CGI program. Then we write a Lua interpreter in PHP. Then we write our website in Lua. Some of those levels are useful. Some of them are redundant. We should embed a Lua interpreter in our webserver and delete two levels of…

Only really relevant if the efficiency is valuable but yeah totally. It’s not just about new abstractions on top of the stack but also new abstractions that replace the middle and bottom of the stacks.

Re: We were wrong about GPUs

#574

Earlier quoted context omitted.

> The right way of cooking RDS in AWS is to go serverless from the start Nitpick, but there is no Serverless for RDS, only Aurora. The two are wildly different in their architecture and performance characteristics. Then there's RDS Multi-AZ Cluster, which is about as confusingly named as they could manage, but I digress. Let's take your stated Minimum ACU of 1 as an example. That gives you 2 GiB of RAM, with "CPU and…

I don't know much about Aurora, there was/is a little too much magic there for my taste, but I feel like once we start with "Postgres- compatible DB", we can't necessarily reason about how things perform under the hood in terms of ordinary Postgres servers. Is there a detailed breakdown of Aurora and its performance/architecture out there? My experience is that AWS is cagey about the details to maintain competitive a…

You can find some re:Invent talks about it, but the level of depth may not be what you want.

The tl;dr is they built a distributed storage system that is split across 3 AZs, each with 2 storage nodes. Storage is allocated in 10 GiB chunks, called protection groups (perhaps borrowing from Ceph’s placement group terminology), with each of these being replicated 6x across the nodes in AZs as mentioned. 4/6 are required for quorum. Since readers are all reading from the same volume, replica lag is typically minimal. Finally, there are fewer / no (not positive honestly; I have more experience with MySQL-compatible Aurora) checkpoints and full page writes.

If you’ve used a networked file system with synchronous writes, you’ll know that it’s slow. This is of course exacerbated with a distributed system requiring 4/6 nodes to ack. To work around this, Aurora has “temporary local storage” on each node, which is a fixed size proportional to the instance size. This is used for sorts that spill to disk, and building secondary indices. This has the nasty side effect that if your table is too large for the local storage, you can’t build new indices, period. AWS will tell you “upsize the instance,” but IMO it’s extremely disingenuous to tout the ability for 128 TiB volumes without mentioning that if a single table gets too big, your schema becomes essentially fixed in place.

Similarly, MySQL normally has something called a change buffer that it uses for updating secondary indices during writes. Can’t have that with Aurora’s architecture, so Aurora MySQL has to write through to the cluster volume, which is slow.

AWS claims that Aurora is anywhere from 3-5x faster than the vanilla versions of the respective DBs, but I have never found this to be true. I’ve also had the goalposts shifted when arguing this point, with them saying “it’s faster under heavy write contention,” but again, I have not found this to be true in practice. You can’t get around data locality. EBS is already networked storage; requiring 4/6 quorum across 3 physically distant AZs makes it even worse.

The 64 TiB limit of RDS is completely arbitrary AFAIK, and is purely to differentiate Aurora. Also, if you have a DB where you need that, and you don’t have a DB expert on staff, you’re gonna have a bad time.

Re: We were wrong about GPUs

#575

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

Wait until LLMS are deployed in edge FPGAs

Re: We were wrong about GPUs

#576
post #426

Earlier quoted context omitted.

> Containers are inherently difficult to sum up in a sentence. Super easy if we talk about Linux. It's a process tree being spawned inside it's own set of kernel namespaces, security measures and a cgroup to provide isolation from the rest of the system.

If someone doesn't understand "container", I'm supposed to expect them to understand all the namespaces and their uses, cgroups, and the nitty gritty of the wimpy security isolation? You are proving my point that it's tough to summarize by using a bunch more terms that are difficult to summarize. Once you recursively expand all the concepts, you will have multiple dense paragraphs, which don't "summarize" anything, b…

If you throw out the Linux tech from my explanation, it would become a general description which holds up even for Windows.

Re: We were wrong about GPUs

#577

Earlier quoted context omitted.

> A Postgres `db.m6g.large` (the cheapest non-burstable instance) runs $114/month for a single AZ, and that's not counting storage or bandwidth. A `db.t4g.medium` runs $47/month, again, not counting storage or bandwidth. This is why numbers do not stack up in the calculations – the premise that the DB has to be provisioned is not the correct one to start off with. The right way of cooking RDS in AWS is to go serverle…

> The right way of cooking RDS in AWS is to go serverless from the start Nitpick, but there is no Serverless for RDS, only Aurora. The two are wildly different in their architecture and performance characteristics. Then there's RDS Multi-AZ Cluster, which is about as confusingly named as they could manage, but I digress. Let's take your stated Minimum ACU of 1 as an example. That gives you 2 GiB of RAM, with "CPU and…

Thanks for the correction, it is Aurora, indeed, although it can be found under the «RDS» console in AWS.

Aurora is actually not a database but is a scalable storage layer that operates over the network and is decoupled from the query engine (compute). The architecture has been used to implement vastly different query engines on top of it (PgSQL, MySQL, DocumentDB – a MongoDB alternative, and Neptune – a property graph database / triple store).

The closest abstraction I can think of to describe Aurora is a VAX/VMS cluster – where the consumer sees a single entity, regardless of size, whilst the scaling (out or back in) remains entirely opaque.

Aurora does not support RDS Proxy for PostgreSQL or its equivalents for other query engine types because it addresses cluster access through cluster endpoints. There are two types of endpoints: one for read-only queries («reader endpoints» in Aurora parlance) and one for read-mutate queries («writer endpoint»). Aurora supports up to 15 reader endpoints, but there can be only one writer endpoint.

Reader endpoints improve the performance of non-mutating queries by distributing the load across read replicas. The default Aurora cluster endpoint always points to the writer instance. Consumers can either default to the writer endpoint for all queries or segregate non-mutating queries to reader endpoints for faster execution.

This behaviour is consistent across all supported query engines, such as PostgreSQL, Neptune, and DocumentDB.

I do not think it is correct to state that Aurora does not use the OS page cache – it does, as there is still a server with an operating system somewhere, despite the «serverless» moniker. In fact, due to its layered distributed architecture, there is now more than one OS page cache, as described in [0].

Since Aurora is only accessible over the network, it introduces unique peculiarities where the standard provisions of storage being local do not apply.

Now, onto the subject of costs. A couple of years ago, an internal client who ran provisioned RDS clusters in three environments (dev, uat, and prod) reached out to me with a request to create infrastructure clones of all three clusters. After analysing their data access patterns, peak times, and other relevant performance metrics, I figured that they did not need provisioned RDS and would benefit from Aurora Serverless instead – which is exactly what they got (unbeknownst to them, which I consider another net positive for Aurora). The dev and uat environments were configured with lower upper ACU's, whilst production had a higher upper ACU configuration, as expected.

Switching to Aurora Serverless resulted in a 30% reduction in the monthly bill for the dev and uat environments right off the bat and nearly a 50% reduction in production costs compared to a provisioned RDS cluster of the same capacity (if we use the upper ACU value as the ceiling). No code changes were required, and the transition was seamless.

Ironically, I have discovered that the AWS cost calculator consistently overestimates the projected costs, and the actual monthly costs are consistently lower. The cost calculator provides a rough estimate, which is highly useful for presenting the solution cost estimate to FinOps or executives. Unintentionally, it also offers an opportunity to revisit the same individuals later and inform them that the actual costs are lower. It is quite amusing.

[0] https://muratbuffalo.blogspot.com/2024/07/understanding-perf...

Re: We were wrong about GPUs

#578

Earlier quoted context omitted.

> The right way of cooking RDS in AWS is to go serverless from the start Nitpick, but there is no Serverless for RDS, only Aurora. The two are wildly different in their architecture and performance characteristics. Then there's RDS Multi-AZ Cluster, which is about as confusingly named as they could manage, but I digress. Let's take your stated Minimum ACU of 1 as an example. That gives you 2 GiB of RAM, with "CPU and…

I don't know much about Aurora, there was/is a little too much magic there for my taste, but I feel like once we start with "Postgres- compatible DB", we can't necessarily reason about how things perform under the hood in terms of ordinary Postgres servers. Is there a detailed breakdown of Aurora and its performance/architecture out there? My experience is that AWS is cagey about the details to maintain competitive a…

There are some details available[0], although they are scarce.

I procured my Aurora intel from a lengthy phone conversation with an exceptionally knowledgeable (and excessively talkative) AWS engineer – who had worked on Aurora – several years ago. The engineer provided detailed explanations of Aurora’s architecture, do's, and dont's as part of our engagement with AWS. The engineer was very proud of AWS’ accomplishments (and I concur that their «something serverless» products are remarkable engineering feats as well as significant cost-saving solutions for me and my clients). The engineer was willing to share many non-sensitive technical details. Generally speaking, a sound understanding of distributed architectures and networks should be sufficient to grasp Aurora Serverless. The actual secret sauce lies in the fine-tuning and optimisations.

[0] https://muratbuffalo.blogspot.com/2024/07/understanding-perf...

Re: We were wrong about GPUs

#579
post #260

Earlier quoted context omitted.

You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it. I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. M…

> Most of what I see is moving sideways, not up the stack. Covering more breadth not height or depth, of abstractions. I don't follow your logic. This comment is so vague. Do you have a specific example?

There are not many languages that for example abstract higher than C# and compile down to it. And whilst we are building abstractions in C# (etc) those are one step up (height), and then there are many of those frameworks, thus many steps sideways (breadth)

Re: We were wrong about GPUs

#580

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Based on the following I think they also meant _how_ the code is running: > The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. I'm a "full full-stack" developer because I understand w…

[deleted]
Post reply on HN