Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EC2 Serial Console

aws.amazon.com

Post reply on HN
Print view View original

71–80 of 121 posts

Re: EC2 Serial Console

#71

Post by herpderperator »

boulos wrote: post #55
commandlinefan wrote: post #8

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

Disclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the b…

I don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0]

[0] https://unix.stackexchange.com/a/153586

Re: EC2 Serial Console

#72

Post by steffan »

koolba wrote: post #20
monocasa wrote: post #5

Earlier quoted context omitted.

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about be…

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

My understanding from the post was that it requires a system root account, not ssh. The former implies the latter, but you could (and usually should) disable root ssh access.

Re: EC2 Serial Console

#73

Post by boulos »

herpderperator wrote: post #71
boulos wrote: post #55

Earlier quoted context omitted.

Disclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the b…

I don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0] [0] https://unix.stackexchange.com/a/153586

The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause.

Amusingly, this finally forced me to find bugs like this one:

https://bugzilla.redhat.com/show_bug.cgi?id=1071290

(All processes started under a remote shell get adjustment -1000, which is basically never shoot me).

There are a few related to setting up the sshd adjustment itself as well.

So, looks like a config problem! Thanks for pointing this out.

Re: EC2 Serial Console

#74

Post by jsolson »

yellowapple wrote: post #65
jsolson wrote: post #62

Earlier quoted context omitted.

I will almost guarantee you two know each other (as someone who knows both of you from Google :)

Pictured: three Google engineers happily doxxing each other (c. 2021, colorized)

I feel like our usernames covered that...

Re: EC2 Serial Console

#75

Post by rantwasp »

commandlinefan wrote: post #8

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

that’s the right answer. unless you are doing something super-specialized or esoteric you should not touch this. and as a matter of fact if you are doing things properly you also don’t have ssh access. remember? cattle not pets!

Re: EC2 Serial Console

#76

Post by sargun »

bigiain wrote: post #46
chousuke wrote: post #17

Earlier quoted context omitted.

Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and th…

They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…

How do you debug broken instances?

Re: EC2 Serial Console

#77

Post by sargun »

boulos wrote: post #55
commandlinefan wrote: post #8

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

Disclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the b…

I did a lot of kernel development with GCP instances. Having the serial port enabled me to use a remote debugger and made it super great.

Re: EC2 Serial Console

#79

Post by x3n0ph3n3 »

sargun wrote: post #76
bigiain wrote: post #46

Earlier quoted context omitted.

They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…

How do you debug broken instances?

Shut them down and mount the volume on a new host.

Re: EC2 Serial Console

#80

Post by Dunedan »

oblio wrote: post #39
ignoramous wrote: post #38

Earlier quoted context omitted.

> ...you can't find docs for it anymore Here are the docs: https://aws.amazon.com/simpledb/

Ah, my bad. But the service is no longer listed in the AWS Console menus.

Because it never was, as it predated the AWS Management Console.
Post reply on HN