Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD/EC2 on C5 instances

daemonology.net

Post reply on HN
Print view View original

11–20 of 45 posts

Re: FreeBSD/EC2 on C5 instances

#11

Post by mbell »

STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

> I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point)

I don't know what you mean by 'late'. Skylake Xeons were delayed and have only been released recently, at least partially. You may be able to get them from vendors like Dell but you still can't go out and buy one anywhere that I'm aware of.

Re: FreeBSD/EC2 on C5 instances

#12

Post by nomadlogic »

kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

NVMe supports SR-IOV much in the same way that NICs do - which i suspect is how AWS is delivering "physical" NICs to VMs currently. So its a pretty safe bet that this is how NVMe devices are being delivered to guest Vm's.

Re: FreeBSD/EC2 on C5 instances

#13

Post by toast0 »

STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets.

Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

Re: FreeBSD/EC2 on C5 instances

#14

Post by cperciva »

kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough.

Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

Re: FreeBSD/EC2 on C5 instances

#15

Post by dastbe »

STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

So I wouldn't say C5s don't work at all with NLB instance target groups - I'm running one right now.

I wouldn't be surprised if you're hitting an edge case with the tighter integration* between NLB's and instance associations. If you haven't already, please do reach out to support.

* from the docs "If you specify targets using an instance ID, the source IP addresses of the clients are preserved and provided to your applications. If you specify targets by IP address, the source IP addresses are the private IP addresses of the load balancer nodes."

Re: FreeBSD/EC2 on C5 instances

#16

Post by joemag »

STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

#2 is expected to work. Can you reach out to me directly (joemag@) and I will make sure we take a look.

Re: FreeBSD/EC2 on C5 instances

#17

Post by jsolson »

cperciva wrote: post #14
kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

It's fun to speculate about how other clouds do things :)

> There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough.

There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible performance versus eliding all VMEXITs via other means, but it has other important characteristics that I suspect AWS very much wants in the c5 family.

> I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

I would expect something along these lines, although I'd be a little surprised if they bothered putting the NVMe bits down in silicon. My personal guess would be a good silicon DMA engine and PCIe interface married to sufficient general-purpose processing (ARM SoC, FPGA, etc.) to keep pace with the NVMe Command/Completion queues.

Regardless of how they've implemented it, the end-to-end result seems to hang together very nicely. Kudos to the team at AWS.

(note: I work on Google Compute Engine's hypervisor; my speculation about AWS really is speculation about how they'd do this — the two companies have very different engineering approaches, so I may be entirely wrong trying to project onto theirs — like I said at the top, it's fun to try :)

Re: FreeBSD/EC2 on C5 instances

#18

Post by kijiki »

jsolson wrote: post #17
cperciva wrote: post #14

Earlier quoted context omitted.

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

It's fun to speculate about how other clouds do things :) > There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible pe…

> I would expect something along these lines, although I'd be a little surprised if they bothered putting the NVMe bits down in silicon.

You and Colin both know they bought Annapurna Labs, right?

We don't have to speculate _that_ much about what is probably going on here...

Re: FreeBSD/EC2 on C5 instances

#19

Post by cperciva »

kijiki wrote: post #18
jsolson wrote: post #17

Earlier quoted context omitted.

It's fun to speculate about how other clouds do things :) > There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible pe…

> I would expect something along these lines, although I'd be a little surprised if they bothered putting the NVMe bits down in silicon. You and Colin both know they bought Annapurna Labs, right? We don't have to speculate _that_ much about what is probably going on here...

That's why I said ARM for the EBS protocol handling rather than MIPS. :-)

I guessed a hardware NVMe interface because that seems like something which could be acquired more or less off-the-shelf, thus minimizing the engineering risks.

Re: FreeBSD/EC2 on C5 instances

#20

Post by cthalupa »

cperciva wrote: post #14
kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

_msw_ mentioned this in the announcement thread (https://news.ycombinator.com/item?id=15640040#15640360):

"but the latest generation EC2 instances offload networking and storage processing to hardware. This is the case for both for instances that use Xen and C5 that uses the new KVM-based hypervisor."

So that seems to confirm an offload to hardware for storage.

Post reply on HN