101–110 of 121 posts
Earlier quoted context omitted.
Who needs nested virtualization when you have bare metal? VM Migration is only for maintenance on GCP -- and customers can't control it, just Google. AWS can hot patch live systems in place without any downtime, so, that's better than a migration (which has a brown out / maintenance period)
> Who needs nested virtualization when you have bare metal? AWS's non-bare-metal systems can boot in ~10s with enough tuning. Their bare-metal systems take tens of minutes to boot. Nested virtualization would allow scaling up and starting new nodes much faster.
Earlier quoted context omitted.
I did the security review of the GCE serial port back when it first came into existence. We probably know each other. The tech industry really is tiny.
Yeah I just checked and you’re both listed as authors on the design doc (as am I, amusingly, though I really don’t remember doing anything particularly useful or significant for it).
Edit: and really the person who did the most work isn’t mentioned here (that’d be up to them)
Earlier quoted context omitted.
You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this:…
> However I still don't understand your comment about the Linux OOM killer wanting to kill sshd "first" (or _ever_ based on these renewed findings!) Can you elaborate? I suspect running low on memory can trigger symptoms that look like sshd failing. sshd gets paged out (or something else you need for a successful login). Un-paging becomes incredibly slow, as there's lots of IO going on from all the paging. Anything g…
In fact, I would guess (especially given all this investigation!) that it’s much more likely that an inaccessible box is just under too much memory pressure for sshd to respond.
Amusingly, the answer is still the same: serial port! :).
Thanks again for all the pointers (to everyone in this thread).
Am I missing something? This is for Nitro instances right, not EC2? Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is
As fguerraz mentioned, modern AWS instance families are basically all powered by Nitro, which refers to the ecosystem around the hypervisor and hardware acceleration cards utilized. https://aws.amazon.com/ec2/nitro/
What's the difference between this and the SSM agent? Can I replace SSM agent configs for this?
I would say they fulfill different purposes. The SSM agent has quite a bit of additional functionality, even within the Session Manager portion. It's more of your solution for online, general day to day access.
Serial console will let you fix issues when you have lost the ability to boot an instance, or network connectivity has failed. When SSH or Session Manager are available, I personally would opt to utilize them over the serial console. But if I have an instance that I can't reach via those, am unable to replace it for whatever reason, and need to bring it back online, serial console would be what I would reach for.
Earlier quoted context omitted.
Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and th…
They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single poin…
What it is actually saying is not that it's a good approach to just throw away servers when they start having problems. What's important is having that ability when it is necessary.
Servers don't just randomly fail. If your instance goes down, sure, your automation will recover your system, but it will still be important to know why it failed, because it may be a symptom of a deeper issue.
It can be due to a hardware issue, but even when that happens, you don't throw away good hardware; you fix it and the server can return to full operation. In the cloud, though, you have no idea what the hardware is doing. Maybe the instance failed because the underlying host failed; maybe it didn't. You should still find out.
I wonder if this is implemented at the hypervisor level (gives you access to the "console") or if this is actually implemented as a serial port.
What would be the difference exactly? The console of a Linux VM is either the emulated serial port or the emulated VGA device + emulated keyboard. I once implemented a tool called virt-dmesg which read out the log_buf from a running Linux kernel (surprisingly useful for those tricky crashes, but difficult from a maintenance point view so the tool is now abandoned). I suppose that's the closest you could get to a "rea…
* http://jdebp.uk./Softwares/nosh/guide/commands/linux-console...