I wonder if this is implemented at the hypervisor level (gives you access to the "console") or if this is actually implemented as a serial port.
EC2 Serial Console
11–20 of 121 posts
Re: EC2 Serial Console
#12This is super useful if you are in the game of building images, or highly tweeking init systems and/or the kernel.
For general consumers, not much value IMHO.
Re: EC2 Serial Console
#13Oh thank god. I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access too anymore. That was... fun. This is a long time coming.
Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.
That's what I was thinking... I mean, this is cool and all, but the only reason I'd ever muck around at this level is if I was actually responsible for the physical hardware and there was a cost associated with replacing it. The whole point of moving to the cloud is to pay somebody else to have to worry about anything I can comprehend needing this for.
Re: EC2 Serial Console
#14I wonder if this is implemented at the hypervisor level (gives you access to the "console") or if this is actually implemented as a serial port.
What would be the difference exactly? The console of a Linux VM is either the emulated serial port or the emulated VGA device + emulated keyboard.
I once implemented a tool called virt-dmesg which read out the log_buf from a running Linux kernel (surprisingly useful for those tricky crashes, but difficult from a maintenance point view so the tool is now abandoned). I suppose that's the closest you could get to a "real" console at the hypervisor level.
Re: EC2 Serial Console
#15Earlier quoted context omitted.
They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about be…
Probably shouldn't have erased their SSH keys then.
I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another.
"Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).
Re: EC2 Serial Console
#16Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.
If something like this happens to an instance, I don't trust the VM's state even after recovery and I'd retire & replace regardless. But I'd love to have serial access to do a root-cause analysis to prevent the issue from occurring in the future.
Re: EC2 Serial Console
#17This seems to be such an obvious feature that I'm surprised they are only adding this now. I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?
Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator.
Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and they can't trust the users.
Re: EC2 Serial Console
#18Earlier quoted context omitted.
Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.
That's what I was thinking... I mean, this is cool and all, but the only reason I'd ever muck around at this level is if I was actually responsible for the physical hardware and there was a cost associated with replacing it. The whole point of moving to the cloud is to pay somebody else to have to worry about anything I can comprehend needing this for.
A lot of medium sized companies moved to the cloud without understanding the tradeoffs. And they have big pockets when it matters.
Re: EC2 Serial Console
#19Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.
Most of the time, that is what I'd suggest to do. However this is like the 4 wheel drive on my truck. I usually don't need it, but when I need it, I really in a bind and glad to have it.
Re: EC2 Serial Console
#20Earlier quoted context omitted.
Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.
They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about be…
To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones.
One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.