Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EC2 Serial Console

aws.amazon.com

Post reply on HN
Print view View original

111–120 of 121 posts

Re: EC2 Serial Console

#111

Post by _msw_ »

pwarner wrote: post #7
petethepig wrote: post #2

This seems to be such an obvious feature that I'm surprised they are only adding this now. I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?

I get this sense they are very conservative in their feature set there? Nested VMs for example are supposed in GCP, Azure and Oracle clouds I think, but not AWS. VM migration too I think exists in GCP at least, but not AWS. It's interesting.

Disclosure: I work for AWS building cloud infrastructure.

I wouldn't assume that VM migration does not exist in AWS. The overall design and implementation of Google's infrastructure somewhat mandated the development live migration support from day one. AWS was designed and built differently, and some types of events that force live migration in GCE do not exist in AWS.

One specific example from Google's VM Live Migration At Scale paper [1] is "Regular maintenance on the power infrastructure in our data centers requires powering down subsets of machines for extended periods of time". The power infrastructure at AWS is designed to be redundant and concurrently maintainable, which removes a significant need for workload mobility within the datacenter.

Personally, I think it was a very good idea to turn the thing that had to be built to launch into a marketed differentiated feature. But that doesn't mean that AWS doesn't have an ability to live migrate some workloads if it is able to do so without disrupting customers, or if it delivers a better experience than alternatives (e.g., instance degrade notices).

[1] https://dl.acm.org/doi/10.1145/3186411.3186415

Re: EC2 Serial Console

#112

Post by roomey »

gregdunn wrote: post #107
roomey wrote: post #89

Am I missing something? This is for Nitro instances right, not EC2? Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is

Disclaimer: I work at AWS, primarily in the compute space, but I'm not speaking in an official capacity. As fguerraz mentioned, modern AWS instance families are basically all powered by Nitro, which refers to the ecosystem around the hypervisor and hardware acceleration cards utilized. https://aws.amazon.com/ec2/nitro/

Thank you very much! I thought nitro referred to the bare metal offering only.

Re: EC2 Serial Console

#113

Post by JoshTriplett »

p0rkbelly wrote: post #102
JoshTriplett wrote: post #70

Earlier quoted context omitted.

> Who needs nested virtualization when you have bare metal? AWS's non-bare-metal systems can boot in ~10s with enough tuning. Their bare-metal systems take tens of minutes to boot. Nested virtualization would allow scaling up and starting new nodes much faster.

If you have a use case for spinning up nodes fast -- what is the use for nested virtualization? Sounds like that should be containers on any underlying single-virtualized layer?

Sometimes you want the safer isolation of virtualization, and you want to spin up individual workload elements in virtual machines.

Re: EC2 Serial Console

#114

Post by bigiain »

chousuke wrote: post #109
bigiain wrote: post #46

Earlier quoted context omitted.

They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…

"cattle vs. pets" misses some important nuance though, especially the way it's usually used to emphasize how you need to architect systems in a cloud environment that way because you have no way of fixing some issues. What it is actually saying is not that it's a good approach to just throw away servers when they start having problems. What's important is having that ability when it is necessary . Servers don't just…

Sure, I totally agree about nuance and the "needing to architect it that way" meaning there. But once you have it architected that way, you then gain the ability to mostly ignore single failures, and only look for "deeper issues" if failures persist.

Even at not-very-high scale, AWS instances _do_ "just randomly fail", at least for all practical interpretations. I don't run anything like FAANG scale, only hundreds of instances rather than thousands or millions, and I see at least a few "random failures" a year (not including spot instances terminating, which I see in clumps every month or so).

I (almost) never try to repair broken a EC2 instance. Wherever I can, they'll be running totally stateless, and I just provision new ones and kill off old ones. I probably won't even bother investigating if it's a rare and singular problem on a known-reliable platform. If one instance wedges and gets replaced, I'll just have a note to investigate if it happens again any time soon. If we get a second failure, we'll go looking in logs and maybe keep and investigate the EBS volume.

For platforms running new-ish code, procedures are different. If we see dead instances after deployments we obviously investigate the new code/config there. But a fair chunk of clients where I am only get 6 or 12 (or even 24) month backend update cycles, if I've got dozens of instances running the same code for months on end and _one_ dies, we just bury it and replace it, and keep a closer eye on the rest of the "herd" for a week or two.

Re: EC2 Serial Console

#115

Post by brian_herman »

yellowapple wrote: post #65
jsolson wrote: post #62

Earlier quoted context omitted.

I will almost guarantee you two know each other (as someone who knows both of you from Google :)

Pictured: three Google engineers happily doxxing each other (c. 2021, colorized)

Can we turn it into an NFT and get more Cereal boxes?

Re: EC2 Serial Console

#117

Post by sargun »

somethingAlex wrote: post #82
sargun wrote: post #76

Earlier quoted context omitted.

How do you debug broken instances?

Typically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.

But what if there is a bug that’s repeatable / pops up on the regular?

Re: EC2 Serial Console

#118

Post by bigiain »

sargun wrote: post #117
somethingAlex wrote: post #82

Earlier quoted context omitted.

Typically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.

But what if there is a bug that’s repeatable / pops up on the regular?

Same as a herd of cattle, if a bunch of them get sick in similar ways you change your process to one where you can find out why. But just one? Shoot it and bury it. Then keep an eye out on the rest in case it's a developing pattern.

Re: EC2 Serial Console

#119

Post by chousuke »

bigiain wrote: post #114
chousuke wrote: post #109

Earlier quoted context omitted.

"cattle vs. pets" misses some important nuance though, especially the way it's usually used to emphasize how you need to architect systems in a cloud environment that way because you have no way of fixing some issues. What it is actually saying is not that it's a good approach to just throw away servers when they start having problems. What's important is having that ability when it is necessary . Servers don't just…

Sure, I totally agree about nuance and the "needing to architect it that way" meaning there. But once you have it architected that way, you then gain the ability to mostly ignore single failures, and only look for "deeper issues" if failures persist. Even at not-very-high scale, AWS instances _do_ "just randomly fail", at least for all practical interpretations. I don't run anything like FAANG scale, only hundreds of…

AWS instances usually "randomly fail" because the underlying hardware has issues. I still don't think it's truly random, but the problem is that you don't really get access to any direct indicators that the host is about to fail before it does. You don't even truly know how old the hardware is, so your risk mitigation strategy has to assume that anything can fail at any time with zero indication of issues beforehand.

When you manage your own physical servers, you have more knowledge of your risk. The actual time of failure will still be random, but if you've been running a host for 5 years straight, you know the risk is growing.

But we're mostly agreeing here. In the scenario where you throw away a "randomly failed" instance, the historical stability is good evidence that it is due to a hardware failure, and you can just replace the instance and move on.

Post reply on HN