Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A $1k AWS mistake

geocod.io

Post reply on HN
Print view View original

241–250 of 281 posts

Re: A $1k AWS mistake

#241

Post by throwawayffffas »

themafia wrote: post #239
ryanjshaw wrote: post #38

As a bootstrapped dev, reading stories like these gives me so much anxiety. I just can’t bring myself to use AWS even despite its advantages.

The documentation is thick but it has a common theme and format to it. So once you get the hang of finding the "juicy bits" you can usually locate them anywhere. The docs do generally warn you of these cases, or have a whole "best practices" section which highlights them directly. The key is, do not make decisions lightly in the cloud, just because something is easy to enable in the UI does not mean it's recommended.…

I am 100% not willing to put that much effort into it.

Actually I am not willing to spend that much time reading "cloud" provider docs and best practices.

What I care about is hosting services and getting the mission done. I value predictable and reasonable costs much more than "flexibility".

In general whatever host I have ever used typically have "cloud" offerings too. So real services go to dedicated hosts, experiments go into 5 buck a month vms.

Re: A $1k AWS mistake

#243

Post by icedchai »

unethical_ban wrote: post #234
icedchai wrote: post #229

Earlier quoted context omitted.

The EC2 needs credentials, but not necessarily a role. If someone is able to compromise an EC2 instance that has unrestricted S3 connectivity (no endpoint policies), they could use their own credentials to exfiltrate data to a bucket not associated with the account.

I'll have to dive in and take a look. I'm not arguing, but here is how I naively see it: It seems there is a gap between "how things are" and "how things should be". "Transiting the internet" vs. "Cost-free intra-region transit" is an entirely different question than "This EC2 has access to S3 bucket X" or "This EC2 does not have access to S3 bucket X". Somewhere, somehow, that fact should be exposed in the design of…

I agree. The real question is why do I need an "VPC endpoint" to save money in the first place?! us-east-1 EC2 isn't actually going over the internet to connect to us-east-1 S3, regardless or whether it's using a NAT gateway or VPC endpoint. AWS knows what routes are on its own network.

Re: A $1k AWS mistake

#245

Post by raw_anon_1111 »

wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

If you are creating a VPC from the console that might be a reasonable default. But any serious implementation is going to be using IAC - like they were - and I would expect to spell out everything explicitly.

Re: A $1k AWS mistake

#246

Post by raw_anon_1111 »

wulfstan wrote: post #131
Spivak wrote: post #126

Earlier quoted context omitted.

The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up. It's annoying because this is by far the more uncommon case for a VPC, but I think it's the rig…

Right, I can appreciate that argument - but then the right thing to do is to block S3 access from AWS VPCs until you have explicitly confirmed that you want to pay the big $$$$ to do so, or turn on the VPC endpoint. A parallel to this is how SES handles permission to send emails. There are checks and hoops to jump through to ensure you can't send out spam. But somehow, letting DevOps folk shoot themselves in the foot…

s3 access is blocked from an EC2 by default unless you give the attached IAM role access to S3.

Then it is still blocked unless you add a NAT gateway or Internet gateway to the VPC and at a route to them.

If you are doing all of this via IAC, you have to take a lot of steps to make this happen. On the other hand, if I’m using an EC2 instance to run an ETL job from data stored on S3, I’m not putting that EC2 instance in a subnet with internet access in the first place. Why would I?

And no you don’t need internet access to access the EC2 instance ftom your computer even without a VPN. You use System Manager Session Manager.

I do the same with lambda - attach then to a VPC without internet access with the appropriate endpoints. Even if they are serving an API, they are still using an API gateway

Re: A $1k AWS mistake

#247

Post by zdc1 »

sgarland wrote: post #99
Waterluvian wrote: post #76

Earlier quoted context omitted.

This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice). I've used AWS for about 10 years and am by no means an expe…

> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I dunno, Aurora’s pricing structure feels an awful lot like that. “What if we made people pay for storage and I/O? And we made estimating I/O practically impossible?”

A year ago I did a back of the napkin calculation and was surprised when I realised Aurora would cost the same or more as my current RDS for Postgres setup. Any discussion with costs inevitably has someone chiming in with a "have you considered Aurora?" and I don't quite understand why it's so loved.

Re: A $1k AWS mistake

#248

Post by raw_anon_1111 »

Spivak wrote: post #165
SOLAR_FIELDS wrote: post #129

Earlier quoted context omitted.

There’s zero reason why AWS can’t pop up a warning if it detects this behavior though. It should clearly explain the implications to the end user. I mean EKS has all sorts of these warning flags it pops up on cluster health there’s really no reason why they can’t do the same here.

I am 100% in agreement, they could even make adding endpoints part of the VPC creation wizard.

How much good does that do if you are using IAC instead of the console - as they are doing?

I wouldn’t even think about doing ClickOps. I’ve worked with AWS for 8 years and I doubt I’ve ever created a resource manually in the console.

Re: A $1k AWS mistake

#249

Post by chillfox »

maccard wrote: post #71
rwmj wrote: post #45

Earlier quoted context omitted.

Wait, what measures you implemented ? How about AWS implements a hard cap, like everyone has been asking for forever?

What does a hard cap look like for EBS volumes? Or S3? RDS? Do you just delete when the limit is hit?

How about something like what runpod does? Shutdown ephemeral resources to ensure there's enough money left to keep data around for some time.

Re: A $1k AWS mistake

#250

Post by array_key_first »

antonvs wrote: post #111
nine_k wrote: post #94

Earlier quoted context omitted.

I'd say that was irony. BTW you can of course self-host k8s, or dokku, or whatnot, and have as easy a deployment story as with the cloud. (But not necessarily as easy a maintenance story for the whole thing.)

> But not as easy a maintenance story That's my whole point. Zero maintenance. For a tinkerer who's focused on the infra, then sure, hosting your own can make sense. But for anyone who's focused on literally anything else, it doesn't make any sense.

> For a tinkerer who's focused on the infra, then sure, hosting your own can make sense.

... or for a big company. I've worked at companies with thousands of developers, and it's all been 'self hosted'. In DCs, so not rinky dink, but yes, and there's a lot of advantages to doing it this way. If you set it up right, it can be much easier for developers to use than AWS.

Post reply on HN