181–190 of 281 posts
Earlier quoted context omitted.
That sounds like an architecture choice? One that would cause less revenue on the AWS side, with a conflicting incentive there.
to be fair, im not sure its a conscious choice, since its not really easy to couple lets say data transfer bytes directly to billing data in real time, and im sure that would also use up a lot of resources. But of course, the incentive to optimize this is not there.
Earlier quoted context omitted.
In theory. Good luck rolling your own version of S3.
You probably don't need it. I see so many people getting price gouged by S3 when it would be orders of magnitude cheaper to just throw the files on a basic HTTP server. I sometimes feel bad using people's services built with S3 as I know my personal usage is costing them a lot of money despite paying them nothing.
These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…
You can transfer from S3 on a single instance usually as fast as the instances NIC--100Gbps+
You'd need a synchronous system that checks quotas before each request and for a lot of systems you'd also need request cancellation (imagine transferring a 5TiB file from S3 and your cap triggers at 100GiB--the server needs to be able to receive a billing violation alert in real time and cancel the request)
I imagine anything capped provided to customers already AWS just estimates and eats the loss
Obviously such a system is possible since IAM/STS mostly do this but I suspect it's a tradeoff providers are reluctant to make
It's all in the docs: https://docs.aws.amazon.com/vpc/latest/privatelink/concepts....
>There is another type of VPC endpoint, Gateway, which creates a gateway endpoint to send traffic to Amazon S3 or DynamoDB. Gateway endpoints do not use AWS PrivateLink, unlike the other types of VPC endpoints. For more information, see Gateway endpoints.
Even the first page of VPC docs: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-ama...
>Use a VPC endpoint to connect to AWS services privately, without the use of an internet gateway or NAT device.
The author of the blog writes:
> When you're using VPCs with a NAT Gateway (which most production AWS setups do), S3 transfers still go through the NAT Gateway by default.
Yes, you are using a virtual private network. Where is it supposed to go? It's like being surprised that data in your home network goes through a router.
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…
The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up. It's annoying because this is by far the more uncommon case for a VPC, but I think it's the rig…
My point is that, architecturally, is there ever in the history of AWS an example where a customer wants to pay for the transit of same-region traffic when a check box exists to say "do this for free"? Authorization and transit/path are separate concepts.
There has to be a better experience.
I get pulled into a fair number of "why did my AWS bill explode?" situations, and this exact pattern (NAT + S3 + "I thought same-region EC2→S3 was free") comes up more often than you’d expect.
The mental model that seems to stick is: S3 transfer pricing and "how you reach S3" pricing are two different things. You can be right that EC2→S3 is free and still pay a lot because all your traffic goes through a NAT Gateway.
The small checklist I give people:
1. If a private subnet talks a lot to S3 or DynamoDB, start by assuming you want a Gateway Endpoint, not the NAT, unless you have a strong security requirement that says otherwise.
2. Put NAT on its own Cost Explorer view / dashboard. If that line moves in a way you didn’t expect, treat it as a bug and go find the job or service that changed.
3. Before you turn on a new sync or batch job that moves a lot of data, sketch (I tend to do this with Mermaid) "from where to where, through what, and who charges me for each leg?" It takes a few minutes and usually catches this kind of trap.
Cost Anomaly Detection doing its job here is also the underrated part of the story. A $1k lesson is painful, but finding it at $20k is much worse.
> AWS charges $0.09 per GB for data transfer out to the internet from most regions, which adds up fast when you're moving terabytes of data. How does this actually work? So you upload your data to AWS S3 and then if you wish to get it back, you pay per GB of what you stored there?
Yes…? Egress bandwidth costs money. Consumer cloud services bake it into a monthly price, and if you’re downloading too much, they throttle you. You can’t download unlimited terabytes from Google Drive. You’ll get a message that reads something like: “Quota exceeded, try again later.” — which also sucks if you happen to need your data from Drive. AWS is not a consumer service so they make you think about the cost dir…
Earlier quoted context omitted.
I'd say that was irony. BTW you can of course self-host k8s, or dokku, or whatnot, and have as easy a deployment story as with the cloud. (But not necessarily as easy a maintenance story for the whole thing.)
> But not as easy a maintenance story That's my whole point. Zero maintenance. For a tinkerer who's focused on the infra, then sure, hosting your own can make sense. But for anyone who's focused on literally anything else, it doesn't make any sense.
Does Amazon refund you for mistakes, or do you have to land on HN frontpage for that to happen?
Earlier quoted context omitted.
No professional engineer uses the AWS console to provision foundational resources like VPC networks.
Especially true now with Claude generating decent terraform code. I was shocked how good it is at knowing AWS gotchas. It also debug connectivity issues almost automagically. While I hate how it writes code I love how it writes terraform.