Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Customers Cannot Escape IPv4

tty.neveragain.de

Post reply on HN
Print view View original

221–230 of 316 posts

Re: AWS Customers Cannot Escape IPv4

#221

Post by forty »

Dylan16807 wrote: post #195
forty wrote: post #184

Earlier quoted context omitted.

I have a reason: we do per IP rate limiting. It's easy enough for IPv4 when the number of IPs is necessarily not too big to fit in a small redis for example, but for IPv6 everyone have at least a /64. I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

When a bunch of households or cell phones are on the same IPv4 do you have any measures to compensate? > Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track. Yes you'd limit by /64 or slightly larger. The live set of IPs shouldn't be very big.

We put limits high enough that it's far enough for any expected usage, including a bunch of users on a single IP. If we see rate limiting happening in practice and it doesn't seem to be an attack, we revisit.

Re: AWS Customers Cannot Escape IPv4

#222

Post by cbreezyyall »

justrealist wrote: post #152
grogenaut wrote: post #134

Earlier quoted context omitted.

I believe the PM was Laura Grit, who was actually a TPM I believe. Laura is a Distinguished Engineer now. She seems to constantly do massive scale projects. IPv4 being a smaller one now. Sadly I can't share some of the big projects she's doing now. I've gotten some sage advice from her on a few occasions that she had time and appreciate it.

> the PM was Laura Grit Talk about nominative determinism...

Imagine never being able to be lazy about anything because the jokes are such a layup.

Re: AWS Customers Cannot Escape IPv4

#223

Post by jonhohle »

discodave wrote: post #137
aranchelk wrote: post #133

Earlier quoted context omitted.

Can you remember what year it was? I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on. Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.

From memory, the regionalization project ran from approx 2014 to 2015 or 2016. There were also other reasons given, like the amount of internal software that used e.g. IPv4 addresses. Also, AWS likes to have 'lots of small things' instead of one big thing (regions, AZs, cells, two pizza teams, no (official) monorepo) so regionalization was part of that. Another big reason for regionalization, other than IPv4 exhausti…

ssh’ing through bastions was such a pain! We used the JMX GUI to review some AMP details from time to time, and port forwarding through the bastions was frowned upon, but our workflow was broken, what were we to do?

IIRC, early on on that project the gateways would get overwhelmed at the volume of traffic they were handling between various VPCs and had to be rolled back several times early on.

Of all the transitions I dealt with at Amazon, snowfort may have been my least favorite (though the ACL/role migration was pretty frustrating as well).

Re: AWS Customers Cannot Escape IPv4

#224

Post by foobiekr »

discodave wrote: post #137
aranchelk wrote: post #133

Earlier quoted context omitted.

Can you remember what year it was? I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on. Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.

From memory, the regionalization project ran from approx 2014 to 2015 or 2016. There were also other reasons given, like the amount of internal software that used e.g. IPv4 addresses. Also, AWS likes to have 'lots of small things' instead of one big thing (regions, AZs, cells, two pizza teams, no (official) monorepo) so regionalization was part of that. Another big reason for regionalization, other than IPv4 exhausti…

This really would not have been true for vendor network gear of the sort AWS had been buying for years by 2014. It's possible that their own switches or the weird fabric they have internally wouldn't have worked with v6, or there were Annapurna NIC ASIC issues, but their primary vendors all would have been fine.

I'm not saying there aren't v6 issues (for some vendors, resource exhaustion might have come into play) or bugs, but there's no way it's that massive a problem. There are huge and complex all v6 networks all over the planet that have more stringent requirements (by law) than AWS DCs.

Re: AWS Customers Cannot Escape IPv4

#225

Post by dilyevsky »

Spivak wrote: post #132
jeroenhd wrote: post #70

Earlier quoted context omitted.

This is one example where it's clear IPv6 isn't the problem, actually. A lot of problems with AWS would disappear if they would just support IPv6 like your average budget ISP does. IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6 IPv4 bridges eas…

> IPv6 just works Only via the herculean efforts of a bunch of people having to literally reinvent the world to deal with it. Everything needs IPv6 support specifically. It's such a mess, if IPv6 has just been identical to IPv4 but with larger addresses we would be on it by now. But no they had to make it their religious crusade to eliminate NAT (and now we have NAT66 so clearly a winner) put IPSec in there which is…

Hey don't forget about NDP and weird ULA thing which is kinda like private but global.

Re: AWS Customers Cannot Escape IPv4

#226

Post by foobiekr »

hughesjj wrote: post #209
discodave wrote: post #137

Earlier quoted context omitted.

From memory, the regionalization project ran from approx 2014 to 2015 or 2016. There were also other reasons given, like the amount of internal software that used e.g. IPv4 addresses. Also, AWS likes to have 'lots of small things' instead of one big thing (regions, AZs, cells, two pizza teams, no (official) monorepo) so regionalization was part of that. Another big reason for regionalization, other than IPv4 exhausti…

You're talking about snowfort, and while ip exhaustion was one reason, it's also an isolation/fault tolerance/security thing.

Indeed, blast radius is a real concern that a lot of folks who try and imitate aws have to learn about the hard way.

Re: AWS Customers Cannot Escape IPv4

#227

Post by jonhohle »

NBJack wrote: post #145
Twirrim wrote: post #126

Earlier quoted context omitted.

I remember the regionalisation, that was "fun" to be on the sidelines for (I was in a newer service that was regionalised from the get-go). I don't remember who the PM was for that one, but I remember that being when I truly came to respect the value that a TPM can add. You're right about the cost and need to replace network equipment being one of the strong reasons why they didn't. Amazon used its own in-house desig…

Were they also by chance considered accelerators for encrypted traffic? I'm not privy to details, but I recall once when a mandate was issued to a Java platform to remove an outdated encryption protocol (mandated by Amazon Infosec). The change was made and rolled out with little fanfare. A few weeks later, a large outage of Amazon Video (which used said platform) occurred on a Friday evening. Root cause? The network…

Maybe not the same story, but there was a sidecar service for encrypting traffic and doing access control and other things in a way that was transparent to the app (like Envoy, but without the mesh and much earlier). The original version was written by (maybe) a single engineer in Erlang. Version two was given to another team and rewritten in Java because. They had never tested at scale and every team I know who went to production with it fell over. There was some company wide deadline, but it was unusable, at the point, and the teams I was working with were gun shy to try it again since it was obvious that the owning team had know idea what the performance characteristics or system requirements were for it.

I think I switched teams before that was resolved and moved to some greenfield work where we didn’t have to worry about scale for a while, but I do believe they eventually figure it out.

Re: AWS Customers Cannot Escape IPv4

#228

Post by Dylan16807 »

Spivak wrote: post #132
jeroenhd wrote: post #70

Earlier quoted context omitted.

This is one example where it's clear IPv6 isn't the problem, actually. A lot of problems with AWS would disappear if they would just support IPv6 like your average budget ISP does. IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6 IPv4 bridges eas…

> IPv6 just works Only via the herculean efforts of a bunch of people having to literally reinvent the world to deal with it. Everything needs IPv6 support specifically. It's such a mess, if IPv6 has just been identical to IPv4 but with larger addresses we would be on it by now. But no they had to make it their religious crusade to eliminate NAT (and now we have NAT66 so clearly a winner) put IPSec in there which is…

> Can you imagine how trivial it would have been if you could support both v4 and v6 by just supporting v6 and having 0::v4addr be literally equivalent to ipv4? It would be more difficult to not support v6.

And how are you supposed to get the packets back when the client has an address outside that range? You still need to add support everywhere, or have NAT gateways into the areas that lack support.

Automatic mapping of IPv4 addresses exists but it requires support infrastructure just as much as any other method of allowing access to IPv4 devices.

Re: AWS Customers Cannot Escape IPv4

#229

Post by ketralnis »

aranchelk wrote: post #133
mabbo wrote: post #104

Many years ago when I was a junior dev at Amazon, there was a massive project internally to split up every internal system into regional versions with limited gateways allowing calls between regions. The reason? We had run out of internal IPv4 addresses. The Principal PM in charge of the "regionalization" effort was asked in a Q&A "why didn't we just switch to IPv6?". Her answer was something along the lines of "The…

Can you remember what year it was? I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on. Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.

> FUD-filled network engineers

FUD sounds like a mean way to say unproven in production

Re: AWS Customers Cannot Escape IPv4

#230

Post by kjs3 »

paulddraper wrote: post #170
aranchelk wrote: post #133

Earlier quoted context omitted.

Can you remember what year it was? I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on. Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.

Right?? How old of a device you would have to get to NOT have IPv6 support? EDIT: But maybe bugs, IDK.

If Amazon is your customer, you fix the bugs; if you're Amazon using your in-house kit, you fix your own bugs whenever you want to. There are plenty of real reasons not to do IPv6, but they are virtually all politics and possibly operational ("we'd have to train our people, and we don't spend money on that"). The idea it was a vendor issue is a BS trope that's been around for at least a decade if not 2.
Post reply on HN