Live data from Hacker News

EC2 Serial Console

aws.amazon.com

51–60 of 121 posts

Re: EC2 Serial Console

#51
post #33

Earlier quoted context omitted.

I see a ton of value when you have instances on a private network with no shell access and want to debug them without setting up a bastion instance.

Yep, this might very well enable whole deployments where there is no SSH access anywhere.

I manage the AWS infrastructure for a medium sized SaaS company and we have no SSH access to any of our ~20 servers. The only access is through AWS Systems Manager Session Manager. Not managing keys is major upside.

Re: EC2 Serial Console

#53
post #7

Earlier quoted context omitted.

I get this sense they are very conservative in their feature set there? Nested VMs for example are supposed in GCP, Azure and Oracle clouds I think, but not AWS. VM migration too I think exists in GCP at least, but not AWS. It's interesting.

Who needs nested virtualization when you have bare metal? VM Migration is only for maintenance on GCP -- and customers can't control it, just Google. AWS can hot patch live systems in place without any downtime, so, that's better than a migration (which has a brown out / maintenance period)

Disclosure: I worked on nested virtualization for GCE.

Even when you offer bare metal, it’s actually still nice to have nested virt! Otherwise, every node has to be a full sized host. So when you have a K8s cluster or similar with a pile of nodes and want to allow some teams to use it (e.g., Android emulator, firecracker, whatever), it’s really nice not to have to say “okay, this group requires full bare metal hosts that they manage themselves”.

tl;dr: nested virt is still a nice to have so that all your infrastructure looks the same.

Edit: Also, you can trigger migration yourself if you want (gcloud compute instances simulate-maintenance-event), but that's mostly to convince yourself that nothing bad will happen.

Re: EC2 Serial Console

#54
post #4
post #3

Oh thank god. I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access too anymore. That was... fun. This is a long time coming.

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

So, a lot of database systems aren't using EBS for many reasons: that's why the local I/O instances exist. And so, yes: you don't build out like that unless but you are prepared to just scrap the instance and rebuild it from WAL logs or have replication or whatever, but knowing "this is a 30 second fix with a serial console" really makes taking all of that cost in stride painful.

Re: EC2 Serial Console

#55

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

Disclosure: I used to work on GCE (and even helped push our serial console access years ago).

That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill.

At this point, without interactive serial port access, you get to replace whatever you’ve got on the box with more logging statements. That’s a totally reasonable approach, but with interactive serial ports you can poke at it and root cause a lot faster.

Edit: Also, Linux seems to always kill sshd first. (Part of this is survivorship bias, of course).

Re: EC2 Serial Console

#56

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

It makes debugging the construction of instances way more annoying, particular if you are dealing with any local networking on the machine. The turnaround time for testing something goes from tens seconds to multiple minutes (at best) without a serial console.

Re: EC2 Serial Console

#57
post #52

I have waited 10 years for this :D. https://forums.aws.amazon.com/thread.jspa?messageID=979374

Wow. Out of curiosity just checked if the other cloud providers have it - both Azure and GCP have this feature already. Azure got it ~ Feb 2020, GCP Feb 20201.

Re: EC2 Serial Console

#58
post #12

This is super useful if you are in the game of building images, or highly tweeking init systems and/or the kernel. For general consumers, not much value IMHO.

I see a ton of value when you have instances on a private network with no shell access and want to debug them without setting up a bastion instance.

AWS's (poorly named) SSM Session Manager service already allows that.

https://docs.aws.amazon.com/systems-manager/latest/userguide...

Re: EC2 Serial Console

#59
post #52

I have waited 10 years for this :D. https://forums.aws.amazon.com/thread.jspa?messageID=979374

Wow. Out of curiosity just checked if the other cloud providers have it - both Azure and GCP have this feature already. Azure got it ~ Feb 2020, GCP Feb 20201.

Disclosure: I used to work on GCE (and was adjacent to the serial port work).

IIRC, we launched interactive serial port access sometime in late 2014. For example, mbrukman answered a SO question on Jan 2, 2015 with connect-to-serial-port [1]. I don’t recall when we gained fancier IAM controls for it, but we’ve had it forever (and I think getting / view only was there at public launch).

[1] https://stackoverflow.com/questions/27734763/how-do-you-acce...

Post reply on HN