Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EC2 Serial Console

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 121 posts

Re: EC2 Serial Console

#91

Post by paulfurtado »

herpderperator wrote: post #81
boulos wrote: post #73

Earlier quoted context omitted.

The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config probl…

You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:…

If you have a large amount of memory, are doing decent amounts of IO, and linux OOMs, the system becomes unresponsive for many minutes before killing any process. At which point, ssh sessions timeout endlessly. A serial console stands a chance.

There is then also any case where you're debugging AMI builds and need to fix grub, or the init system without waiting 20 minutes for a new AMI build each time.

Also, the existing console log feature in AWS is insultingly not real time. It doesn't typically update at all unless you're within minutes of boot or trigger a reboot and it only buffers something like 4kb so a reboot can easily fully replace the logs. This really sucks when you're trying to get the debug console output, so this feature finally solves that.

Re: EC2 Serial Console

#92

Post by [deleted] »

roomey wrote: post #89

Am I missing something? This is for Nitro instances right, not EC2? Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is

[deleted]

Re: EC2 Serial Console

#93

Post by petercooper »

monocasa wrote: post #5
koolba wrote: post #4

Earlier quoted context omitted.

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about be…

You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.

I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.

Re: EC2 Serial Console

#94

Post by kubanczyk »

bigiain wrote: post #46
chousuke wrote: post #17

Earlier quoted context omitted.

Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and th…

They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…

I'd use it to troubleshoot quirky AMIs which simply do not boot in a specific setting no matter how many times I try.

Anything more esoteric than "a normal Ubuntu" can have a bug, e.g. it hangs with three network interfaces or similar.

Re: EC2 Serial Console

#95

Post by herpderperator »

paulfurtado wrote: post #91
herpderperator wrote: post #81

Earlier quoted context omitted.

You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:…

If you have a large amount of memory, are doing decent amounts of IO, and linux OOMs, the system becomes unresponsive for many minutes before killing any process. At which point, ssh sessions timeout endlessly. A serial console stands a chance. There is then also any case where you're debugging AMI builds and need to fix grub, or the init system without waiting 20 minutes for a new AMI build each time. Also, the exis…

Why would linux trigger OOM if you have a large amount of memory available? Or, what did you mean by "large amount of memory"?

Also, why would an SSH session, which is entirely in memory, time out because of I/O thrashing? You can disconnect the hard drive that sshd and/or the OS is running from and your SSH connections to that machine won't break. If you run some commands that aren't cached in memory you'll naturally get critical I/O errors, but it won't cause a disconnect on the SSH layer.

Re: EC2 Serial Console

#96

Post by michaelt »

herpderperator wrote: post #81
boulos wrote: post #73

Earlier quoted context omitted.

The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config probl…

You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:…

> However I still don't understand your comment about the Linux OOM killer wanting to kill sshd "first" (or _ever_ based on these renewed findings!) Can you elaborate?

I suspect running low on memory can trigger symptoms that look like sshd failing.

sshd gets paged out (or something else you need for a successful login). Un-paging becomes incredibly slow, as there's lots of IO going on from all the paging. Anything garbage-collected starts running GC constantly, using 100% CPU.

Then your attempt to SSH times out - and with no access to list running processes, one naturally concludes sshd has failed.

Re: EC2 Serial Console

#97

Post by oblio »

Dunedan wrote: post #80
oblio wrote: post #39

Earlier quoted context omitted.

Ah, my bad. But the service is no longer listed in the AWS Console menus.

Because it never was, as it predated the AWS Management Console.

S3 also predates the console, what kind of argument is that?

Re: EC2 Serial Console

#98

Post by tremon »

monocasa wrote: post #22
koolba wrote: post #20

Earlier quoted context omitted.

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.

This is what I do at home (kvm serial console on sulogin with root's password locked), but I would never consider that for a multi-tenant cloud provider.

A simple "rely on IAM for access control" doesn't match with my idea of defense in depth.

Re: EC2 Serial Console

#100

Post by whalesalad »

sciurus wrote: post #58
whalesalad wrote: post #21

Earlier quoted context omitted.

I see a ton of value when you have instances on a private network with no shell access and want to debug them without setting up a bastion instance.

AWS's (poorly named) SSM Session Manager service already allows that. https://docs.aws.amazon.com/systems-manager/latest/userguide...

It requires an agent.
Post reply on HN