Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A $1k AWS mistake

geocod.io

Post reply on HN
Print view View original

121–130 of 281 posts

Re: A $1k AWS mistake

#121

Post by belter »

wiether wrote: post #56
belter wrote: post #31

[flagged]

There's nothing to gain in punching down They made a mistake and are sharing it for the whole word to see in order to help others avoid making it. It's brave. Unlike punching down.

This has nothing about punching down. Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. Its like when Google bought thousands of servers without ECC memory, but felt they were so smart they could not resist telling the world how bad that was and writing a paper about it...Or they could have hired some real hardware engineers from IBM or Sun...

Re: A $1k AWS mistake

#122

Post by maccard »

__s wrote: post #79
maccard wrote: post #71

Earlier quoted context omitted.

What does a hard cap look like for EBS volumes? Or S3? RDS? Do you just delete when the limit is hit?

It's a system people opt into, you can do something like ingress/egress blocked, & user has to pay a service charge (like overdraft) before access opened up again. If account is locked in overdraft state for over X amount of days then yes, delete data

I can see the "AWS is holding me ransom" posts on the front page of HN already.

Re: A $1k AWS mistake

#123

Post by wat10000 »

maccard wrote: post #71
rwmj wrote: post #45

Earlier quoted context omitted.

Wait, what measures you implemented ? How about AWS implements a hard cap, like everyone has been asking for forever?

What does a hard cap look like for EBS volumes? Or S3? RDS? Do you just delete when the limit is hit?

A cap is much less important for fixed costs. Block transfers, block the ability to add any new data, but keep all existing data.

Re: A $1k AWS mistake

#124

Post by agwa »

tlaverdure wrote: post #100

Abolish NAT Gateways. Lean on gateway endpoints, egress only internet gateways with IPv6, and security groups to batten down the hatches. All free.

Now that AWS charges for public IPv4 addresses, is it still free if you need to access IPv4-only hosts?

Re: A $1k AWS mistake

#125

Post by perching_aix »

withinboredom wrote: post #103
wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

Or just run bare metal + garage and call it a day.

I personally prefer to just memorize the data and recite it really quickly on-demand.

Only half-joking. When something grossly underperforms, I do often legitimately just pull up calc.exe and compare the throughput to the number of employees we have × 8 kbit/sec [0], see who would win. It is uniquely depressing yet entertaining to see this outperform some applications.

[0] spherical cow type back of the envelope estimate, don't take it too seriously; assumes a very fast 200 wpm speech, 5 bytes per word, and everyone being able to independently progress

Re: A $1k AWS mistake

#126

Post by Spivak »

wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up.

It's annoying because this is by far the more uncommon case for a VPC, but I think it's the right way to structure, permissions and access in general. S3, the actual service, went the other way on this and has desperately been trying to reel it back for years.

Re: A $1k AWS mistake

#127

Post by otterley »

scotty79 wrote: post #95
Waterluvian wrote: post #76

Earlier quoted context omitted.

This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice). I've used AWS for about 10 years and am by no means an expe…

> I think that the lack of half-decent cost controls is not intentionally malicious It wasn't when the service was first created. What's intentionally malicious is not fixing it for years. Somehow AI companies got this right form the get go. Money up front, no money, no tokens. It's easy to guess why. Unlike hosting infra bs, inference is a hard cost for them. If they don't get paid, they lose (more) money. And sendi…

> Somehow AI companies got this right form the get go. Money up front, no money, no tokens.

That’s not a completely accurate characterization of what’s been happening. AI coding agent startups like Cursor and Windsurf started by attracting developers with free or deeply discounted tokens, then adjusted the pricing as they figure out how to be profitable. This happened with Kiro too[1] and is happening now with Google’s Antigravity. There’s been plenty of ink spilled on HN about this practice.

[1] disclaimer: I work for AWS, opinions are my own

Re: A $1k AWS mistake

#128

Post by philipwhiuk »

dabiged wrote: post #13

I made the same mistake and blew $60k. I have never understood why the S3 endpoint isn't deployed by default, except to catch people making this exact mistake.

Yeah imagine the conversation:

"I'd like to spend the next sprint on S3 endpoints by default"

"What will that cost"

"A bunch of unnecessary resources when it's not used"

"Will there be extra revenue?"

"Nah, in fact it'll reduce our revenue from people who meant to use it and forgot before"

"Let's circle back on this in a few years"

Re: A $1k AWS mistake

#129

Post by SOLAR_FIELDS »

Spivak wrote: post #126
wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up. It's annoying because this is by far the more uncommon case for a VPC, but I think it's the rig…

There’s zero reason why AWS can’t pop up a warning if it detects this behavior though. It should clearly explain the implications to the end user. I mean EKS has all sorts of these warning flags it pops up on cluster health there’s really no reason why they can’t do the same here.

Re: A $1k AWS mistake

#130

Post by MrDarcy »

coredog64 wrote: post #118
wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

If you use the AWS console, it's a tick box to include this.

No professional engineer uses the AWS console to provision foundational resources like VPC networks.
Post reply on HN