Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets per Second Limitations in EC2

bluematador.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Packets per Second Limitations in EC2

#11

Post by toast0 »

Note that if your traffic hits the ec2 connection tracking security groups, you will also hit per instance limits on the number of tracked connections [1]. As far as I know, they don't come out and say they have a limit on the number of tracked connections, but they do, and it scales by instance type -- better to adjust your rules so the traffic is allowed in a stateless manner.

I don't know, but wouldn't be surprised if connection tracked packets are more limited than packets that aren't tracked.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne...

Re: Packets per Second Limitations in EC2

#12

Post by greglindahl »

ttul wrote: post #3

Amazon also limits DNS queries - probably in a well meaning attempt to prevent DNS amplification attacks from originating within AWS. And I mean DNS queries across their network whether or not they hit Amazon's DNS servers. This is _any_ port 53 UDP traffic. https://www.sparkpost.com/blog/undocumented-limit-dns-aws/

I wonder if this is related to connection tracking?

Re: Packets per Second Limitations in EC2

#13

Post by Rapzid »

I guess technically it is, but I hesitate to call the need for PPS limitations in the DC as "over subscribing".

Connect a single server to a network and it's oversubscribed. That's a bit hyperbolic, but even some beefy networks can be seriously burdened by just a single server spamming UDP packets without some sort of QoS.. Especially if they are bypassing user space and using the kernel to just replicate a bunch of packets onto the wire :)

I'm probably a bit biased from having spent time setting up linux TC on xen hypervisors for this very reason; and I think we even settled at 50k pps for the per vm limit too..

Re: Packets per Second Limitations in EC2

#14

Post by edoceo »

ra1n85 wrote: post #10
rbranson wrote: post #2

The (undocumented) PPS limitation on EC2 instance types before they added SR-IOV NICs is around 150K PPS. If you had your own full machine — usually the top size of a given instance class, but no guarantee — this would be pretty consistent. But it was a shared resource. This made running memcache clusters really painful on EC2, given that they’d easily get limited by packet throughput before CPU or bandwidth. With mo…

???? Where are you getting these numbers? For what instance types? For what protocols? This is just wrong.

GP said it was undocumented (I presume empirical, but would like more details)

Do you have a more accurate dataset? What are your observations?

I'm assuming this affects loads of HN readers and I too am interested in what the facts are.

Side note: you may be getting downvoted because a source, or other details are lacking. I too get downvoted for posts that lack these details.

Re: Packets per Second Limitations in EC2

#15

Post by blazespin »

Rapzid wrote: post #13

I guess technically it is, but I hesitate to call the need for PPS limitations in the DC as "over subscribing". Connect a single server to a network and it's oversubscribed. That's a bit hyperbolic, but even some beefy networks can be seriously burdened by just a single server spamming UDP packets without some sort of QoS.. Especially if they are bypassing user space and using the kernel to just replicate a bunch of…

They should still publish the limits. If they are reasonable, customers won't mind.

Re: Packets per Second Limitations in EC2

#17

Post by greglindahl »

toast0 wrote: post #11

Note that if your traffic hits the ec2 connection tracking security groups, you will also hit per instance limits on the number of tracked connections [1]. As far as I know, they don't come out and say they have a limit on the number of tracked connections, but they do, and it scales by instance type -- better to adjust your rules so the traffic is allowed in a stateless manner. I don't know, but wouldn't be surprise…

That sure sounds like it's being processed by the standard Linux firewall. In which case, yeah, if you have (my favorite example) a web crawler operating on the general web, you'll hit serious limits.

Re: Packets per Second Limitations in EC2

#20

Post by _kp6z »

A more plausible explanation is that the xen networking path is simply expensive, the intel VFs are limited by queue count and silicon (i40e isn't a great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been abandoned by most silicon vendors and have a tragic history. It's simply hard to make NPUs work right at attractive price/power/performance and at high speed versus fixed function scatter/gather I/O units coupled with general purpose CPUs running software network stacks. The only benefit Annapurna gives EC2 over a software device model is a hard security boundary of effectively another computer inside the computer for Nitro metal as a service. I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time.

Here is a demonstration of a software stack that can scale toward hardware limits without relying on a particular vendor https://www.slideshare.net/SeanChittenden/freebsd-vpc-introd.... This approaches 100G line rate for large packets which is what it was optimized for. I don't know PPS at low packet size but do know what would be required to optimize that use case and it could be done pretty quickly.

Post reply on HN