Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EC2 Serial Console

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 121 posts

Re: EC2 Serial Console

#81

Post by herpderperator »

boulos wrote: post #73
herpderperator wrote: post #71

Earlier quoted context omitted.

I don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0] [0] https://unix.stackexchange.com/a/153586

The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config probl…

You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted.

Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:

  server ~ # grep oom_score_adj /usr/sbin/sshd
  grep: /usr/sbin/sshd: binary file matches
...which is fascinating and clever. That's when I checked the source code linked at "[0]". I now finally have an answer as to why I've seen dmesg memory stat dumps display different oom_score_adj values for sshd. I always thought _something_ was smart enough to know that we don't want to risk killing sshd, but I didn't know what that _something_ was. It turns out it was the daemon itself.

  Oct 13 23:20:38 server kernel: Mem-Info:
  ...
  Oct 13 23:20:38 server kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
  Oct 13 23:20:38 server kernel: [ 2455]    60  2455  1778653   273114     710      10        0             0 mysqld
  ...
  Oct 13 23:20:38 server kernel: [12085]   207 12085    22574      673      32       3        0             0 tlsmgr
  Oct 13 23:20:38 server kernel: [ 4238]     0  4238     9234      518      20       3        0         -1000 systemd-udevd
  Oct 13 23:20:38 server kernel: [12278]     0 12278    88107     5597     136       4        0             0 apache2
  Oct 13 23:20:38 server kernel: [17222]     0 17222  1258983   142035     505       8        0             0 qemu-system-x86
  ...
  Oct 13 23:20:38 server kernel: [21069]     0 21069     5033      487      14       4        0             0 bash
  Oct 13 23:20:38 server kernel: [15935]     0 15935     7081      487      16       3        0         -1000 sshd
  ...
In retrospect it makes a lot of sense, especially considering sshd runs as root -- it has complete ability to do that. And it's not like anything else would know the importance of sshd except for itself.

However I still don't understand your comment about the Linux OOM killer wanting to kill sshd "first" (or _ever_ based on these renewed findings!) Can you elaborate?

[0] https://github.com/openssh/openssh-portable/blob/e51dc7fab61...

Re: EC2 Serial Console

#82

Post by somethingAlex »

sargun wrote: post #76
bigiain wrote: post #46

Earlier quoted context omitted.

They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…

How do you debug broken instances?

Typically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.

Re: EC2 Serial Console

#83

Post by rrdharan »

MikeHalcrow wrote: post #61
boulos wrote: post #59

Earlier quoted context omitted.

Disclosure: I used to work on GCE (and was adjacent to the serial port work). IIRC, we launched interactive serial port access sometime in late 2014. For example, mbrukman answered a SO question on Jan 2, 2015 with connect-to-serial-port [1]. I don’t recall when we gained fancier IAM controls for it, but we’ve had it forever (and I think getting / view only was there at public launch). [1] https://stackoverflow.com/q…

I did the security review of the GCE serial port back when it first came into existence. We probably know each other. The tech industry really is tiny.

Yeah I just checked and you’re both listed as authors on the design doc (as am I, amusingly, though I really don’t remember doing anything particularly useful or significant for it).

Re: EC2 Serial Console

#85

Post by somethingAlex »

herpderperator wrote: post #69

Was this really not available before? It is such a basic requirement for virtual machines. Other providers like Linode have had this since 2009 if not earlier.

I find it interesting that half of the comments indicate that people don't really understand why this would be needed and the other half are surprised that is hasn't been offered before.

Personally, I'm in the first camp. I'm used to taking instances for granted on the rare occasion that a low-level issue arises and just promoting a replica or trashing it if it's stateless.

I'm assuming the use case where you care about fixing the type of issues this feature helps debug is fairly esoteric?

Re: EC2 Serial Console

#86

Post by kalleboo »

monocasa wrote: post #22
koolba wrote: post #20

Earlier quoted context omitted.

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.

The article says "the only requirement is that the root account has been assigned a password, as this is the one you will use to log in", but obviously that will only apply to the default Amazon Linux AMI configuration

Re: EC2 Serial Console

#87

Post by ec109685 »

herpderperator wrote: post #81
boulos wrote: post #73

Earlier quoted context omitted.

The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config probl…

You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:…

That code is 12 years old: https://github.com/openssh/openssh-portable/commit/c8802aac2...

So maybe the parent post was remembering a time before then.

Re: EC2 Serial Console

#88

Post by jake_morrison »

I would have loved to have that when working on building AMIs for custom Linux systems, e.g. https://github.com/cogini/buildroot_ec2 and https://github.com/cogini/nerves_system_ec2

I spent a lot of time looking at console screenshots of machines that would not boot and iterating to figure out the problem.

Re: EC2 Serial Console

#89

Post by roomey »

Am I missing something? This is for Nitro instances right, not EC2?

Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is

Re: EC2 Serial Console

#90

Post by fguerraz »

roomey wrote: post #89

Am I missing something? This is for Nitro instances right, not EC2? Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is

All modern instances families (regardless of their type) are powered by Nitro.
Post reply on HN