Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Amazon Prime Day 2023 AWS Bill

lastweekinaws.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: The Amazon Prime Day 2023 AWS Bill

#41

Post by orochimaaru »

mrbonner wrote: post #28

It’s not a surprise for me to hear that Amazon is still a heavy user of RDBMS all these years even after the so-called Rolling Stone project to get rid of Oracle DB in 2015. If Amazon can use RDBMS for their scale, I’m just furious when folks jumping up and down screaming in top of their lungs “Why do we use Postgres and not (insert some random NoSQL engine here)?” My response so far is calmly ask another question “W…

Your application use cases should dictate the database choice - eg consistency needed, access patterns, data normalization, reliability, etc.

Re: The Amazon Prime Day 2023 AWS Bill

#42

Post by fiddlerwoaroof »

greatpostman wrote: post #25
CamperBob2 wrote: post #14

Earlier quoted context omitted.

Hot take: Amazon's search UX is so terrible that it not only wastes near-endless amounts of customer time and patience, but their own bandwidth as well.

They’ve a/b tested it to death

I wonder if Amazon has overfitted and/or a/b tester itself into a bad local optima. It’s pretty hard for me to believe that their current website really is as good as their data indicates.

Re: The Amazon Prime Day 2023 AWS Bill

#43

Post by figassis »

rqtwteye wrote: post #22
version_five wrote: post #13

Amazon Prime Day event resulted in an incremental 163 petabytes of EBS storage capacity allocated – generating a peak of 15.35 trillion requests and 764 petabytes of data transfer per day. The main thing that strikes me is how (seemingly) inefficient everything is. What do they possibly need this amount of data for in selling stuff? Are they taking high-def video of every customer as they browse for something to buy?…

It seems to me that a lot of modern architectures store the same data in multiple places. The systems I see proposed in my company probably need often 10 times more space than the actual data we have because they copy and cache a lot of stuff.

Microservices requires denormalizing data across tables and dbs. There’s a cost to how many microservices you build.

Re: The Amazon Prime Day 2023 AWS Bill

#44

Post by jeffbee »

The amount of mail alone is bonkers. If we assume that half of this traffic went to the big operators, Google and Microsoft, each of them would have observed a noticeable traffic bump, 10s of 1000s of requests per second on average all day. It is fun to think about how these systems are interconnected and how they affect each other.

Re: The Amazon Prime Day 2023 AWS Bill

#45

Post by thayne »

version_five wrote: post #13

Amazon Prime Day event resulted in an incremental 163 petabytes of EBS storage capacity allocated – generating a peak of 15.35 trillion requests and 764 petabytes of data transfer per day. The main thing that strikes me is how (seemingly) inefficient everything is. What do they possibly need this amount of data for in selling stuff? Are they taking high-def video of every customer as they browse for something to buy?…

A lot of that was certainly just for the root volumes of all those ec2 instances (how much exactly is hard to know without more details). Which of course would have duplicate copies of the various base images for the VMs.

Although, that does bring up the question of why AWS doesn't have a way to share a single read-only volume across multiple ec2 instances in the same availability zone. In many workloads there isn't any need to write to disk.

Re: The Amazon Prime Day 2023 AWS Bill

#46

Post by thenewarrakis »

version_five wrote: post #13

Amazon Prime Day event resulted in an incremental 163 petabytes of EBS storage capacity allocated – generating a peak of 15.35 trillion requests and 764 petabytes of data transfer per day. The main thing that strikes me is how (seemingly) inefficient everything is. What do they possibly need this amount of data for in selling stuff? Are they taking high-def video of every customer as they browse for something to buy?…

I think the EBS numbers are "double counting". Most of the other services in the list are using EBS under the hood, so I wouldn't be surprised if this number includes stuff like the Aurora instances, CloudTrail events, SQS events, etc that are also included.

Also, it specifically says "incremental capacity allocated", not necessarily used. Keep in mind that every EC2 instance launched also means new EBS storage is allocated. The article also estimates that 50 million EC2 instances were used for Prime Day. If you assume that half of these were newly created to support the surge of Prime Day, 25 million instances using up 160 PB of storage is only 6 gigabytes per instance, which definitely seems in the realm of possibility.

Re: The Amazon Prime Day 2023 AWS Bill

#47

Post by schlarpc »

thayne wrote: post #45
version_five wrote: post #13

Amazon Prime Day event resulted in an incremental 163 petabytes of EBS storage capacity allocated – generating a peak of 15.35 trillion requests and 764 petabytes of data transfer per day. The main thing that strikes me is how (seemingly) inefficient everything is. What do they possibly need this amount of data for in selling stuff? Are they taking high-def video of every customer as they browse for something to buy?…

A lot of that was certainly just for the root volumes of all those ec2 instances (how much exactly is hard to know without more details). Which of course would have duplicate copies of the various base images for the VMs. Although, that does bring up the question of why AWS doesn't have a way to share a single read-only volume across multiple ec2 instances in the same availability zone. In many workloads there isn't…

There kind of is, but it's not really made for that use case so there's a bunch of caveats (it's read/write, has a limited max number of attachments, io1/2 required, can't be the boot volume): https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu...

Re: The Amazon Prime Day 2023 AWS Bill

#48

Post by thayne »

fiddlerwoaroof wrote: post #42
greatpostman wrote: post #25

Earlier quoted context omitted.

They’ve a/b tested it to death

I wonder if Amazon has overfitted and/or a/b tester itself into a bad local optima. It’s pretty hard for me to believe that their current website really is as good as their data indicates.

IME a/b tests are often run by people with little to no knowledge of statistics or experimental procedure. It is pretty easy to end up backing bad decisions with data when you don't completely understand the data.

Re: The Amazon Prime Day 2023 AWS Bill

#49

Post by thayne »

nonameiguess wrote: post #10
mokarma wrote: post #2

Naive question: What are they using EBS for? It seems unnecessary given all the Databases. Is that just local caching for EC2's?

At minimum, root volumes for the VMs. Theoretically, you could load immutable machine images from the network and run entirely off of in-memory filesystems if you persist nothing past instance shutdown (similar to how extremely cautious people might run Tails booted off USB on a laptop with no hard drive), but that won't actually save cost since memory is more expensive than disk anyway.

I don't think you can even technically do that in AWS. I don't think there is any way to detach the root volume from a running instance, or use an immutable network image to boot from. However, for many server workloads, operating entirely would be reasonable. Often you just need the operating system kernel and your server software, and maybe a monitoring agent. And all of that will be loaded in memory anyway.

Re: The Amazon Prime Day 2023 AWS Bill

#50

Post by ndriscoll »

madrox wrote: post #11
rurp wrote: post #9

> $102 million in infrastructure spend for an event that brought in over $12.7 billion in sales isn’t the worst return on investment that companies could make — by a landslide! Well it's not amazing if your margin's are tiny, as they are in many industries (such as retail). Plus this was almost certainly architected by some of the foremost AWS experts in the world. It's verrrry easy to spend vastly more than was stri…

Important to remember that, before you could burst your infrastructure in the cloud, sites simply went offline in events like this. You took actively lost revenue in those cases.

Or you could just design your architecture to not perform trillions of database requests for hundreds of millions of sales.

The listing data is almost static and should almost fit in RAM (the hot set probably does. Apparently Amazon has ~350M listings. A 24TB RAM server could give ~68kB/listing, and probably only a small fraction is hot). Since you'll need multiple servers anyway, you could shard on products and definitely fit things in RAM. 375 million sales even if condensed into 1 hour would only be 104k/second. A single db server should be able to handle the cart/checkout. Assuming ~10M page views/second, a couple racks of servers should be able to handle it.

The ad/tracking infrastructure surely can't account for the 1000x disparity in resource usage.

Post reply on HN