Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets per Second Limitations in EC2

bluematador.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Packets per Second Limitations in EC2

#21

Post by samstave »

edoceo wrote: post #14
ra1n85 wrote: post #10

Earlier quoted context omitted.

???? Where are you getting these numbers? For what instance types? For what protocols? This is just wrong.

GP said it was undocumented (I presume empirical, but would like more details) Do you have a more accurate dataset? What are your observations? I'm assuming this affects loads of HN readers and I too am interested in what the facts are. Side note: you may be getting downvoted because a source, or other details are lacking. I too get downvoted for posts that lack these details.

I took him saying "just wrong" as being "im not OK with this", but i could be incorrect.

However, with that said, i have always found that calling your rep and asking about specific un documented limits is the fastest way to get to the bottom of per-instance/account/vpc/whatever limits.

Just as there are limits that can be changed if you agree, in writing, that you will be financially responsible for whatever the impact is (e.g. when you could tell them that you wanted spot price limits adjusted for you to be able to better bid above the scaling factors that were in place(not sure if this is still the case))

Some limits are global and cant be changed/negotiated, but other undocumented limits....

Re: Packets per Second Limitations in EC2

#22

Post by discodave »

_kp6z wrote: post #20

A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I…

Interesting theories on the EC2/Annapurna situation.

Do GCP, Azure, or any other cloud providers offer 100G networking?

Re: Packets per Second Limitations in EC2

#23

Post by toredash »

greglindahl wrote: post #17
toast0 wrote: post #11

Note that if your traffic hits the ec2 connection tracking security groups, you will also hit per instance limits on the number of tracked connections [1]. As far as I know, they don't come out and say they have a limit on the number of tracked connections, but they do, and it scales by instance type -- better to adjust your rules so the traffic is allowed in a stateless manner. I don't know, but wouldn't be surprise…

That sure sounds like it's being processed by the standard Linux firewall. In which case, yeah, if you have (my favorite example) a web crawler operating on the general web, you'll hit serious limits.

There is a limit of you have a Security Group attached with a rule that is -not- 0.0.0.0/0. So for anything that is public / heavy utilized, the recommendation is to open the service up to 0.0.0.0/0.

Re: Packets per Second Limitations in EC2

#24

Post by cperciva »

_kp6z wrote: post #20

A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I…

I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time.

Interestingly, the ENA driver has #defines for speeds up to 400 Gbps.

My guess as to why EC2 instances are limited to 25 Gbps is that it's a matter of balancing overprovisioning and the need to avoid having a single instance eat too much of a rack's bandwidth. I don't know how much bandwidth they have going to each rack, but there's a limit to how much it makes sense to provision; if typical bandwidth is on the order of 10 Gbps per rack (say, 80 instances pushing 125 Mbps on average) then you might want to provision 200 Gbps/rack and limit each instance to 25 Gbps rather than provisioning 1 Tbps/rack and limiting each instance to 100 Gbps.

(Numbers above are completely invented; I don't have any internal knowledge of how Amazon's networks or datacenters are set up.)

Re: Packets per Second Limitations in EC2

#25

Post by john37386 »

greglindahl wrote: post #12
ttul wrote: post #3

Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries across their network whether or not they hit Amazon's DNS servers. This is _any_ port 53 UDP traffic. https://www.sparkpost.com/blog/undocumented-limit-dns-aws/

I wonder if this is related to connection tracking?

By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.

Re: Packets per Second Limitations in EC2

#26

Post by john37386 »

EC2 throttles everything by default and PPS is no exception.

What can you do when your system needs more bandwidth, cpu, ram or any other kind of resources?

You can either scale vertically... which is not bad at beginning of a project, but sooner or later you will hit the ultimate limit.

Or

You can scale horizontally. Which means that you have enough nodes or instances to bypass those limits and make sure your projects grow well over time.

Netflix runs on EC2 and they probably generate billions of PPS from Amazon. For sure it's several millions PPS a d they seem to not hit the limits mentionned in the article.

Re: Packets per Second Limitations in EC2

#27

Post by madaxe_again »

_kp6z wrote: post #20

A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I…

Agreed - I read this and saw XPS being the culprit writ large.

AWS aren’t alone in this, and actually do pretty darn well compared to their competition - we had a nightmarish time a few years back with exactly this with a VPS provider - half of every second the traffic to the memcached cluster would just stop. Turned out they’d set hard limits on packets/sec to avoid oversaturating the host, so the advertised Gbps interconnect was actually 50Mbps when you saturated the packet scheduler.

Re: Packets per Second Limitations in EC2

#28

Post by patrickg_zill »

Eventually people will realize how AWS overcharges for what they deliver. But of course there's nothing wrong with pricing yourself higher than the lowest cost option...

This is perhaps already being seen in a piecemeal fashion as people compare eg S3 storage prices with other companies' prices.

Re: Packets per Second Limitations in EC2

#29

Post by spydum »

john37386 wrote: post #25
greglindahl wrote: post #12

Earlier quoted context omitted.

I wonder if this is related to connection tracking?

By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.

I think you may be mistaken. Security Groups are stateful: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit...

As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne...

I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).

Re: Packets per Second Limitations in EC2

#30

Post by spydum »

john37386 wrote: post #26

EC2 throttles everything by default and PPS is no exception. What can you do when your system needs more bandwidth, cpu, ram or any other kind of resources? You can either scale vertically... which is not bad at beginning of a project, but sooner or later you will hit the ultimate limit. Or You can scale horizontally. Which means that you have enough nodes or instances to bypass those limits and make sure your projec…

What is it you think netflix runs on AWS? Content distribution is served from their Open Connect CDN, not AWS.. last I understood, most of Netflix cloud workloads were analytical/DWH, and services.. Not generally billions of PPS, and certainly not to single instances.
Post reply on HN