Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD/EC2 on C5 instances

daemonology.net

Post reply on HN
Print view View original

31–40 of 45 posts

Re: FreeBSD/EC2 on C5 instances

#31

Post by STRML »

toast0 wrote: post #13
STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets. Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

Worse still, we spent weeks on the phone with AWS insisting there was no throttling - just to find out there was throttling. And that would have been discoverable if a single engineer had looked at CloudWatch.

Re: FreeBSD/EC2 on C5 instances

#32

Post by STRML »

tty7 wrote: post #10
STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

You have given me a whole other perspective to an aws customer. I spend a lot on aws (70k/m +) but id be happy if i had core2duo cpus! What sort of work do you have that requires the latest generation ? Or more why do you want the latest ? Id expect aws to always be behind the ball - are they the right platform for you? Gce is interesting, im moving half of my infra over there - but again its not really about their h…

Our particular use case involves a few processes that are heavily serial, while yet memory-intensive; the fastest possible processor would be a boon to us (for instance, if someone would guarantee an overclocked Xeon, we'd take it in a heartbeat for almost any price).

AWS is indeed behind the curve more often than not, but they also have some amazing hosted products. That was a bigger deal in 2014 than it is now (Kubernetes will eat the world) but it's still a reliable offering, and reliability is our most important criteria.

Agreed re: being multi-cloud in the end. It's the only responsible choice above a certain scale.

Re: FreeBSD/EC2 on C5 instances

#33

Post by STRML »

joemag wrote: post #16
STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

#2 is expected to work. Can you reach out to me directly (joemag@) and I will make sure we take a look.

There's an open support ticket, but I've reached out as well.

Re: FreeBSD/EC2 on C5 instances

#34

Post by aliguori »

kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

> Hopefully Amazon will disclose more details.

We will have some more details on how this all works at re:Invent in a couple weeks.

Re: FreeBSD/EC2 on C5 instances

#35

Post by otterley »

STRML wrote: post #31
toast0 wrote: post #13

Earlier quoted context omitted.

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets. Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

Worse still, we spent weeks on the phone with AWS insisting there was no throttling - just to find out there was throttling. And that would have been discoverable if a single engineer had looked at CloudWatch.

Do you have Enterprise Support? If so, this is a very surprising result.

Re: FreeBSD/EC2 on C5 instances

#36

Post by otterley »

toast0 wrote: post #13
STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets. Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

> They also have hidden limits on the number of incoming tcp connections you can have.

Do you have proof of this?

Re: FreeBSD/EC2 on C5 instances

#37

Post by otterley »

STRML wrote: post #30
_msw_ wrote: post #24

Earlier quoted context omitted.

Does the updated documentation at http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitorin... explain the CloudWatch metrics?

That's more helpful. That's a lot of special cases - and thought - to put into each time we look at a C5-attached volume as opposed to any other. It also does not include any mention of Average Read Latency and Average Write Latency, which appears to be incorrect in all dimensions (average, min, max, sum).

We've found that CloudWatch is useful for some system and I/O related metrics (i.e., the from the EBS SAN's point of view) but less useful for others (i.e., metrics from the VM's point of view). It's worthwhile to install a monitoring agent on the VM that can collect I/O latency and other statistics there too. We use Datadog, but there are lots of options out there (collectd etc.).

Re: FreeBSD/EC2 on C5 instances

#38

Post by toast0 »

otterley wrote: post #36
toast0 wrote: post #13

Earlier quoted context omitted.

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets. Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

> They also have hidden limits on the number of incoming tcp connections you can have. Do you have proof of this?

No, I had an experiment I was tried to run on EC2 in 2013, and ran into this. It was very clear though. Established connections would plateau, and then no more tcp syns would arrive to the ec2 host, unless a connection was closed. At the time the limit was 5000 connections on the micro instance -- from my notes, we got an allocation for hi1.4xlarge and I know we hit the limit there too, but I don't recall what the limit was. This was a very simple TCP proxy, using Linux kernel ipmasq; this uses significantly less memory than HAProxy, although with a lot less features.

We had a very excited account rep because of where I work, but he was only barely able to confirm the limits were there, he wasn't able to get them raised or removed, nor could he tell us the limits by machine type.

There's a thread from 18 months ago on the aws forums [1], where an aws rep more or less confirms, but again provides no information

[1] https://forums.aws.amazon.com/thread.jspa?threadID=231806

Re: FreeBSD/EC2 on C5 instances

#39

Post by _msw_ »

STRML wrote: post #30
_msw_ wrote: post #24

Earlier quoted context omitted.

Does the updated documentation at http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitorin... explain the CloudWatch metrics?

That's more helpful. That's a lot of special cases - and thought - to put into each time we look at a C5-attached volume as opposed to any other. It also does not include any mention of Average Read Latency and Average Write Latency, which appears to be incorrect in all dimensions (average, min, max, sum).

This is a topic that we're continuing to iterate on between EBS, CloudWatch, and the AWS console team for metrics. When a newly introduced behavior makes step function changes in graphs displayed in the AWS console it doesn't meet the principle of least astonishment.

We're also continuing to investigate the reported latency in the console. Because this is a derived metric from VolumeTotal{Read,Write}Time and Volume{Read,Write}Ops, there may be a miscalculation happening due to the change in dimensions.

Re: FreeBSD/EC2 on C5 instances

#40

Post by cperciva »

_msw_ wrote: post #39
STRML wrote: post #30

Earlier quoted context omitted.

That's more helpful. That's a lot of special cases - and thought - to put into each time we look at a C5-attached volume as opposed to any other. It also does not include any mention of Average Read Latency and Average Write Latency, which appears to be incorrect in all dimensions (average, min, max, sum).

This is a topic that we're continuing to iterate on between EBS, CloudWatch, and the AWS console team for metrics. When a newly introduced behavior makes step function changes in graphs displayed in the AWS console it doesn't meet the principle of least astonishment. We're also continuing to investigate the reported latency in the console. Because this is a derived metric from VolumeTotal{Read,Write}Time and Volume{R…

Not sure if this is related, but when I was getting EFS working last year I noticed that CloudWatch graphs were often complete nonsense due to EFS not logging zeroes for idle filesystems but CloudWatch treating this as "missing data" rather than "implied zeroes".
Post reply on HN