Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS in 2025: Stuff you think you know that's now wrong

lastweekinaws.com

Post reply on HN
Print view View original

161–170 of 284 posts

Re: AWS in 2025: Stuff you think you know that's now wrong

#161

Post by baby_souffle »

gigatexal wrote: post #67
gchamonlive wrote: post #66

Earlier quoted context omitted.

S3 + cloudfront is also incredibly popular so you can just find recipes for automating that in any technology you want, Terraform, ansible, plain bash scripts, Cloudformation (god forbid)

Yeah holy crap why is cloud formation so terrible?

> Yeah holy crap why is cloud formation so terrible?

I can't confirm it, but I suspect that it was always meant to be a sales tool.

Every AWS announcement blog has a "just copy this JSON blob, and paste it $here to get your own copy of the toy demo we used to demonstrate in this announcement blog" vibe to it.

Re: AWS in 2025: Stuff you think you know that's now wrong

#162

Post by dragonwriter »

gigatexal wrote: post #67
gchamonlive wrote: post #66

Earlier quoted context omitted.

S3 + cloudfront is also incredibly popular so you can just find recipes for automating that in any technology you want, Terraform, ansible, plain bash scripts, Cloudformation (god forbid)

Yeah holy crap why is cloud formation so terrible?

Because its an old early IaC language, but it works and lots depends on it, so instead of dumping or retooling it, AWS keeps it around as a compilation target, while pushing other solutions (years ago, the SAM transform on top of it, more recently CDK) as the main thing for people to actually use directly.

Re: AWS in 2025: Stuff you think you know that's now wrong

#163

Post by cmcarthur »

biimugan wrote: post #160

> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots. From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well. From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share)…

This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue.

Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention.

For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.

Re: AWS in 2025: Stuff you think you know that's now wrong

#165

Post by JoshTriplett »

otterley wrote: post #82
JoshTriplett wrote: post #78

Earlier quoted context omitted.

That's fascinating! I hadn't found that in the documentation; everything seems to steer people towards PrivateLink, not gateway endpoints. Would you recommend using VPC Gateway even on a public VPC that has an Internet gateway (note: not a NAT gateway)? Or only on a private VPC or one with a NAT gateway?

I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet. Plus they eliminate the need for NAT Gateway traversal for requests that originate from private subnets.

> I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet.

Fascinating. What's the advantage of doing that?

Re: AWS in 2025: Stuff you think you know that's now wrong

#166

Post by UltraSane »

philistine wrote: post #157
danudey wrote: post #145

Earlier quoted context omitted.

The most likely explanation is that they used a tape robot, such as the one seen here: https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_ro... Which is basically exactly what you described but the picker is a robot. Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards i…

I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.

For truly write once read never data tape is the optimal storage method. It is exactly what the LTO standard was designed to do and it does it very well. You can be confident that you will be able to read every bit of data from a 30 year old tape, probably even 50 years old. It has the lowest bit error rate of any technology I am aware of. LTO-9 is better than 1 uncorrectable bit error in 10^20 user bits, which is 1 bit error in 12.5 exabytes. There is also the substantial advantage that tapes on a shelf are completely immune to ransomware. As a sysadmin I get that warm fuzzy feeling when critical data is backed up on a good LTO tape library.

Re: AWS in 2025: Stuff you think you know that's now wrong

#167

Post by danpalmer »

Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated.

That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.

Re: AWS in 2025: Stuff you think you know that's now wrong

#168

Post by count »

UltraSane wrote: post #166
philistine wrote: post #157

Earlier quoted context omitted.

I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.

For truly write once read never data tape is the optimal storage method. It is exactly what the LTO standard was designed to do and it does it very well. You can be confident that you will be able to read every bit of data from a 30 year old tape, probably even 50 years old. It has the lowest bit error rate of any technology I am aware of. LTO-9 is better than 1 uncorrectable bit error in 10^20 user bits, which is 1…

Call it bad luck, but I’ve never had a fully successful restore. Drives eat tapes, drives are damaged and write bad data, robot arms die or malfunction. Tapes have NEVER worked for me. SANs and remote disk though, rock solid.

That said, I don’t miss any of that stuff, gimme S3 any day :)

Re: AWS in 2025: Stuff you think you know that's now wrong

#169

Post by hinkley »

lijok wrote: post #85
simonw wrote: post #76

Earlier quoted context omitted.

I wonder if it's where old S3 hard drives go to die? Presumably AWS have the world's single largest collection of used storage devices - if you RAID them up you can probably get reliable performance out of them for Glacier?

You don’t raid old drives as it creates cascading failures because recovering from a failed drive adds major wear to other drives

Only if you have low redundancy. RAIDZ is better about this isn’t it? And Backblaze goes a lot farther. They just decommission the rack when it hits the limit for failed disks, and the files on the cluster are stored on m of n racks, so adding a rack and “resilvering” doesn’t even require scanning the entire cluster, just m/n of it.

Re: AWS in 2025: Stuff you think you know that's now wrong

#170

Post by dabiged »

UltraSane wrote: post #166
philistine wrote: post #157

Earlier quoted context omitted.

I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.

For truly write once read never data tape is the optimal storage method. It is exactly what the LTO standard was designed to do and it does it very well. You can be confident that you will be able to read every bit of data from a 30 year old tape, probably even 50 years old. It has the lowest bit error rate of any technology I am aware of. LTO-9 is better than 1 uncorrectable bit error in 10^20 user bits, which is 1…

As someone who does tape recovery on very very old tape I largely concur with this with a couple of caveats.

1. Do not encrypt your tapes if you want the data back in 30/50 years. We have had so many companies lose encryption keys and turn their tapes into paperweights because the company they bought out 17 years ago had poor key management.

2. The typical failure case on tape is physical damage not bit errors. This can be via blunt force trauma (i.e. dropping, or sometimes crushing) or via poor storage (i.e. mould/mildew).

3. Not all tape formats are created equal. I have seen far higher failure rates on tape formats that are repeatedly accessed, updated, ejected, than your old style write once, read none pattern.

Post reply on HN