Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A $1k AWS mistake

geocod.io

Post reply on HN
Print view View original

131–140 of 281 posts

Re: A $1k AWS mistake

#131

Post by wulfstan »

Spivak wrote: post #126
wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up. It's annoying because this is by far the more uncommon case for a VPC, but I think it's the rig…

Right, I can appreciate that argument - but then the right thing to do is to block S3 access from AWS VPCs until you have explicitly confirmed that you want to pay the big $$$$ to do so, or turn on the VPC endpoint.

A parallel to this is how SES handles permission to send emails. There are checks and hoops to jump through to ensure you can't send out spam. But somehow, letting DevOps folk shoot themselves in the foot (credit card) is ok.

What has been done is the monetary equivalent of "fail unsafe" => "succeed expensively"

Re: A $1k AWS mistake

#132

Post by kccqzy »

ndriscoll wrote: post #114
kccqzy wrote: post #108

Earlier quoted context omitted.

Reserving the cost until the end of the billing cycle is super unfriendly for spiky traffic and spiky resource usage. And yet one of the main selling points of the cloud is elasticity of resources. If your load is fixed, you wouldn’t even use the cloud after a five minute cost comparison. So your solution doesn’t work for the intended customers of the cloud.

It works just fine. No reason you couldn't adjust your billing cap on the fly. I work in a medium size org that's part of a large one, and we have to funnel any significant resource requests (e.g. for more EKS nodes) through our SRE teams anyway to approve. Actual spikey traffic that you can't plan for or react to is something I've never heard of, and believe is a marketing myth. If you find yourself actually trying…

I think I am having some misunderstanding about exactly how this cost control works. Suppose that a company in the transportation industry needs 100 CPUs worth of resources most of the day and 10,000 CPUs worth of resources during morning/evening rush hours. How would your reserved cost proposal work? Would it require having a cost cap sufficient for 10,000 CPUs for the entire day? If not, how?

Re: A $1k AWS mistake

#133

Post by tlaverdure »

agwa wrote: post #124
tlaverdure wrote: post #100

Abolish NAT Gateways. Lean on gateway endpoints, egress only internet gateways with IPv6, and security groups to batten down the hatches. All free.

Now that AWS charges for public IPv4 addresses, is it still free if you need to access IPv4-only hosts?

Yeah not free if you definitely need IPv4. AWS has been adding a lot more IPv6 support to their services so hopefully the trend continues in AWS and the broader industry. You can probably get pretty far though if your app doesn't have hard requirements to communicate with IPv4 only hots.

Re: A $1k AWS mistake

#134

Post by abigail95 »

plantain wrote: post #34
embedding-shape wrote: post #28

Earlier quoted context omitted.

"Premium bandwidth" which AWS/Amazon markets to less understanding developers is almost a scam. By now, software developers think data centers, ISPs and others part of the peering on the internet pay per GB transferred, because all the clouds charge them like that.

Try a single threaded download from Hetzner Finland versus eu-north-1 to a remote (i.e. Australia) destination and you'll see premium bandwidth is very real. Google Cloud Storage significantly more so than AWS. Sure you can just ram more connections through the lossy links from budget providers or use obscure protocols, but there's a real difference. Whether it's fairly priced, I suspect not.

I just tested it and TCP gets the maximum expected value given the bandwidth delay product from a server in Falkenstein to my home in Australia, from 124 megabits on macOS to 940 megabits on Linux.

Can you share your tuning parameters on each host? If you aren't doing exactly the same thing on AWS as you are on Hetzner you will see different results.

Bypassing the TCP issue I can see nothing indicating low network quality, a single UDP iperf3 pass maintains line rate speed without issue.

Edit: My ISP peers with Hetzner, as do many others. If you think it's "lossy" I'm sure someone in network ops would want to know about it. If you're getting random packet loss across two networks you can have someone look into it on both ends.

Re: A $1k AWS mistake

#135

Post by SOLAR_FIELDS »

pjc50 wrote: post #54
merpkz wrote: post #12

> AWS charges $0.09 per GB for data transfer out to the internet from most regions, which adds up fast when you're moving terabytes of data. How does this actually work? So you upload your data to AWS S3 and then if you wish to get it back, you pay per GB of what you stored there?

Nine cents per gigabyte feels like cellphone-plan level ripoff rather than a normal amount for an internet service. And people wonder why Cloudflare is so popular, when a random DDoS can decide to start inflicting costs like that on you.

I don’t mind the extortionate pricing if it’s upfront and straightforward. fck-nat does exist. What I do mind is the opt out behavior that causes people to receive these insane bills when their first, most obvious expectation is that traffic within a data center stays within that data center and doesn’t flow out to the edge of it and back in. That is my beef with the current setup.

But “security” people might say. Well, you can be secure and keep the behavior opt out, but you should be able to have an interface that is upfront and informs people of the implications

Re: A $1k AWS mistake

#136

Post by wulfstan »

MrDarcy wrote: post #130
coredog64 wrote: post #118

Earlier quoted context omitted.

If you use the AWS console, it's a tick box to include this.

No professional engineer uses the AWS console to provision foundational resources like VPC networks.

Yes, this. You lock it into Terraform or some equivalent.

And ok, this is a mistake you will probably only make once - I know, because I too have made it on a much smaller scale, and thankfully in a cost-insensitive customer's account - but surely if you're an infrastructure provider you want to try to ensure that you are vigilantly removing footguns.

Re: A $1k AWS mistake

#137

Post by gbear605 »

otterley wrote: post #127
scotty79 wrote: post #95

Earlier quoted context omitted.

> I think that the lack of half-decent cost controls is not intentionally malicious It wasn't when the service was first created. What's intentionally malicious is not fixing it for years. Somehow AI companies got this right form the get go. Money up front, no money, no tokens. It's easy to guess why. Unlike hosting infra bs, inference is a hard cost for them. If they don't get paid, they lose (more) money. And sendi…

> Somehow AI companies got this right form the get go. Money up front, no money, no tokens. That’s not a completely accurate characterization of what’s been happening. AI coding agent startups like Cursor and Windsurf started by attracting developers with free or deeply discounted tokens, then adjusted the pricing as they figure out how to be profitable. This happened with Kiro too[1] and is happening now with Google…

I think you’re talking about a different thing? The bad practice from AWS et al is that you post-pay for your usage, so usage can be any amount. With all the AI things I’ve seen, either: - you prepay a fixed amount (“$200/mo for ChatGPT Max”) - you deposit money upfront into a wallet, if the wallet runs out of cash then you can’t generate any more tokens - it’s free!

I haven’t seen any of the major model providers have a system where you use as many tokens as you want and then they bill you, like AWS has.

Re: A $1k AWS mistake

#138

Post by kikimora »

MrDarcy wrote: post #130
coredog64 wrote: post #118

Earlier quoted context omitted.

If you use the AWS console, it's a tick box to include this.

No professional engineer uses the AWS console to provision foundational resources like VPC networks.

Especially true now with Claude generating decent terraform code. I was shocked how good it is at knowing AWS gotchas. It also debug connectivity issues almost automagically. While I hate how it writes code I love how it writes terraform.

Re: A $1k AWS mistake

#139

Post by iso1631 »

withinboredom wrote: post #103
wulfstan wrote: post #80

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…

Or just run bare metal + garage and call it a day.

Or colocate your bare metal in two or three data centres for resilience against environmental issues and single supplier.

Re: A $1k AWS mistake

#140

Post by duped »

Waterluvian wrote: post #76
Havoc wrote: post #37

These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…

This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice). I've used AWS for about 10 years and am by no means an expe…

> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.

It's someone in a Patagonia vest trying to avoid getting PIP'd.

Post reply on HN