Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD/EC2 on C5 instances

daemonology.net

Post reply on HN
Print view View original

21–30 of 45 posts

Re: FreeBSD/EC2 on C5 instances

#21

Post by jsolson »

kijiki wrote: post #18
jsolson wrote: post #17

Earlier quoted context omitted.

It's fun to speculate about how other clouds do things :) > There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible pe…

> I would expect something along these lines, although I'd be a little surprised if they bothered putting the NVMe bits down in silicon. You and Colin both know they bought Annapurna Labs, right? We don't have to speculate _that_ much about what is probably going on here...

I do, hence my speculation about putting the NVMe in firmware instead of hardware :)

(Colin's speculation in a peer reply is also reasonable -- personally I've seen enough errata in "off the shelf" IP to shudder at the idea of anything in silicon that doesn't have to be, but fundamentally I'm a SWE, so that would be my take, wouldn't it)

Re: FreeBSD/EC2 on C5 instances

#22

Post by jsolson »

cthalupa wrote: post #20
cperciva wrote: post #14

Earlier quoted context omitted.

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

_msw_ mentioned this in the announcement thread ( https://news.ycombinator.com/item?id=15640040#15640360 ): "but the latest generation EC2 instances offload networking and storage processing to hardware. This is the case for both for instances that use Xen and C5 that uses the new KVM-based hypervisor." So that seems to confirm an offload to hardware for storage.

Yes, sorry, I fully expect that's what they're doing -- I was really writing two replies there --

1) You don't have to pass PCIe through to hardware to get performance that makes it look like you have.

2) We know they're offloading though, so here's what I think it looks like.

Re: FreeBSD/EC2 on C5 instances

#23

Post by cperciva »

jsolson wrote: post #21
kijiki wrote: post #18

Earlier quoted context omitted.

> I would expect something along these lines, although I'd be a little surprised if they bothered putting the NVMe bits down in silicon. You and Colin both know they bought Annapurna Labs, right? We don't have to speculate _that_ much about what is probably going on here...

I do, hence my speculation about putting the NVMe in firmware instead of hardware :) (Colin's speculation in a peer reply is also reasonable -- personally I've seen enough errata in "off the shelf" IP to shudder at the idea of anything in silicon that doesn't have to be, but fundamentally I'm a SWE, so that would be my take, wouldn't it)

The way I see it, everything has errata... but if you're taking something off the shelf, it's more likely that someone else already found them. :-)

Re: FreeBSD/EC2 on C5 instances

#24

Post by _msw_ »

STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

Does the updated documentation at http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitorin... explain the CloudWatch metrics?

Re: FreeBSD/EC2 on C5 instances

#25

Post by _msw_ »

jsolson wrote: post #17
cperciva wrote: post #14

Earlier quoted context omitted.

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

It's fun to speculate about how other clouds do things :) > There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible pe…

And I could speculate on hypervisor bypass in Andromeda 2.1 ;-)

Re: FreeBSD/EC2 on C5 instances

#26

Post by jsolson »

_msw_ wrote: post #25
jsolson wrote: post #17

Earlier quoted context omitted.

It's fun to speculate about how other clouds do things :) > There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. There's a wide spectrum between "emulated" and "real hardware, exposed via PCI passthrough". Passing through to PCI hardware, in and of itself, gains you very little in terms of absolute guest-visible pe…

And I could speculate on hypervisor bypass in Andromeda 2.1 ;-)

Indeed! I didn't leave too much to the imagination with my replies on the original post[0], though.

Honestly, I'm mostly curious about how much of "KVM" you're running that's stock, how much is modified, and how much of the userland is running on the far side of PCIe rather than in host ring3 (particularly given "C5 instances are built using a new light-weight hypervisor, which provides practically all of the compute and memory resources to customers’ instances.").

[0]: Especially this one: https://news.ycombinator.com/item?id=15641391

Re: FreeBSD/EC2 on C5 instances

#27

Post by ksec »

cperciva wrote: post #3
X86BSD wrote: post #2

Colin is one of the many smart wizards we are lucky to have in FreeBSD land. Well done Colin!

I can't claim much credit here. I haven't made anything work; all I did was figure out what didn't work and let the right people know.

That is basically what a Project Manager does, and from my perspective it is a lot more valuable then what can been seen form both inside and outside.

Re: FreeBSD/EC2 on C5 instances

#28

Post by cperciva »

ksec wrote: post #27
cperciva wrote: post #3

Earlier quoted context omitted.

I can't claim much credit here. I haven't made anything work; all I did was figure out what didn't work and let the right people know.

That is basically what a Project Manager does, and from my perspective it is a lot more valuable then what can been seen form both inside and outside.

Fair enough. Maybe I should say that I didn't demonstrate any special talents here. What I do could have been done by anyone in the FreeBSD project, but I ended up managing the FreeBSD/EC2 platform by accident and now I'm the obvious person to keep on managing the platform.

Re: FreeBSD/EC2 on C5 instances

#29

Post by jchw »

cperciva wrote: post #14
kijiki wrote: post #7

> YOU'RE BUILDING A HARDWARE FRONT-END TO EBS? You guys are insane! It seems more likely that they've put in a software device model of NVME as a replacement for the BlkBack software device model that the BlkFront driver talked to. Not much different than the software e1000 NIC that xen/qemu already supports. That said, with their virtualizable Annapurna wonder-NIC, they could be doing it in "hardware", though even i…

There's absolutely no way that they would get the performance I'm seeing from an emulated disk. We're talking to real hardware, exposed via PCI passthrough. Now, exactly what form that hardware takes is an open question. I would assume it's something like "NVME interface hardware" + "ARM CPU which implements the EBS protocol" + "25 GbE PHY", but that guess is based solely on "that's how I would design it".

Throughput is one thing, but what does the latency look like? I am no expert, but my assumption is that latency is the bigger problem with network-attached storage.

Re: FreeBSD/EC2 on C5 instances

#30

Post by STRML »

_msw_ wrote: post #24
STRML wrote: post #6

While the new C5 instances are certainly welcome - I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point) - we have encountered a number of show-stopping problems that point to this project being just a bit too ambitious. To name a few: 1. EBS volumes attached to C5 instances show completely bogus CloudWatch metrics, over an order of magnitu…

Does the updated documentation at http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitorin... explain the CloudWatch metrics?

That's more helpful. That's a lot of special cases - and thought - to put into each time we look at a C5-attached volume as opposed to any other. It also does not include any mention of Average Read Latency and Average Write Latency, which appears to be incorrect in all dimensions (average, min, max, sum).
Post reply on HN