Live data from Hacker News

Show HN: SadServers – Test your Linux troubleshooting skills

sadservers.com

61–70 of 133 posts

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#61

Earlier quoted context omitted.

I haven't fully grokked this yet, but one trick I've used in the past to get around limits is AWS Organizations, creating a sub-account per property. A bit more setup but can keep things cleaner administratively.

AWS will raise limits if you ask. Increasing EC2 instance limits is usually a quick turn around.

At least for the tests I've done on a small startup recently, they've also implemented some automatic quota increases for EC2. I ran commands that would have (or did) eclipsed my quota, and got an email that my quotas were bumped a few minutes later.

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#62

>Practice for your next SRE/DevOps interview. Are SREs and DevOps tasked with administration of operating systems?

Both SRE and DevOps are essentially evolved sysadmin roles. The DevOps philosophy is cross-functional and many sysadmins have adopted a DevOps approach. The latest edition of the classic sysadmin book "The Practice of System and Network Administration" is now centered around DevOps.

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#63

Based on your architecture diagram it looks like you're spinning up an instance per-user? As you're probably finding now, you will hit AWS limits quickly. You might instead want to have a smaller pool of (larger) servers that you run co-resident VMs on with https://firecracker-microvm.github.io/ . That will avoid account limits and also keep your AWS costs more predictable.

Just run them in Linux VMs with WASM, on the users' browsers. Make them all pay for it with higher utility bills and greater wear & tear on their hardware.

trollface.jpg

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#64
post #29

I'd suggest integrating https://bellard.org/jslinux/ and running the VM in the browser if you can - then you can scale without running out of resources.

Thanks, I've been looking at WASM, for ex https://github.com/snaplet/postgres-wasm/tree/main/packages/... , it would certainly simplify everything to "download a fat file".

Have you seen https://copy.sh/v86/ ? It doesn't run as fast as jslinux but is BSD Licensed, on Github, and supports resuming the VM from a snapshot.

https://github.com/copy/v86

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#65
post #19
post #15

Can't get to the first problem because of HN hug but anyway there are fake ways to "solve" it like renaming the logfile (what they test for solved is provided).

Depends on how the broken program writes to the log. If it does while true; do echo hello >> bad.log; done Then renaming bad.log will not solve the challenge.

Replace it with a symlink to /dev/null! Or /dev/full if we feel like it.

(Yes, these are bad solutions, since the instructions explicitly said to stop the process which is writing.)

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#66

Very cool! This reminds me of the ops challenge @ Slack. I'm not sure if they still do this, but the SRE/platform infra interview used to involve a VM running a malfunctioning LAMP stack. You'd get SSH access to the VM, then submit a diagnostic report of what was broken (and how you fixed it). Reminded me of how Red Hat used to run their certification test (RHCE). I probably still have the live CDs for my RHCE laying…

I've had interviews like that in the past, and really enjoyed them. Much better than "Draw an architecture diagram for how you'd handle a serverless IoT application" - where you lose points, silenly, because you didn't pick something the interviewer expected you to do.

Usually a simple combination of immutable files, SELinux policies, and types in configuration files were enough for most of the challenges. Though now and again you'd find they'd given you a server with packages removed, or not yet installed.

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#67
post #24

Earlier quoted context omitted.

Why are you afraid of this? My org has run a hands-on technical exam with a stack of linux admin basics (I won't enumerate them here because people do their research) but they are based on real problems we've had and the feedback is overwhelmingly "this was one of the best technical interviews I've ever had." We ask the engineer who is proctoring the interview to think about the following question: Would you want to…

If you give the person you're interviewing access to the same tools they'd have in a regular day on the job (Google, manpages, etc.), I'd say that's a fair and probably relatively enjoyable interview. Rejecting someone because they can't recall the correct netstat syntax doesn't seem like good hiring practice, but I assume in good faith that's not what you meant :)

Yeah, I google, tealdear, "--help", and manpage anything I don't use at least once a week, every time. Usually I don't remember them otherwise, and if I think I do, I don't trust my memory that well. Only exception is if I remember enough to be able to ctrl+r them out of shell history faster than I can do those things—and actually, for some of those, I do use them often, but couldn't possibly tell you how because I only run a couple commands 99% of the time and always pull them out of history unless it's one of the rare exceptional cases—I couldn't rsync for a particular outcome without consulting a reference, to save my life, even though I use it often.

And usually you only use a fairly small set of tools that often, in any job, and which set will depend on the employer, how things are set up, and what exactly you're doing.

Oh and somehow I get "-r" versus "-R" for "recursive" wrong almost every time, even for commands I type almost daily, unless I check first. It's weird. If tools could get on the same damn page about which means "recursive", that'd be great.

TL;DR I do have a pretty good idea what I'm doing, but look like an absolute idiot if anyone watches me do it. Much worse, even, if I know they're watching and we're not in some kind of relatively high-trust relationship (so, definitely not in an interview setting).

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#68

My only feedback is that this is unrealistic because today developers wouldn’t try to debug something, they’d just destroy the instance, push a commit and hope it fixed something infra related then recreate it. Why would you need to understand how something works? Just use containers. /s

If most developers can't debug a VM, then anyone who can will be able to charge a premium. If you have a proficiency in ops, remember that the next time you negotiate a compensation package. [Edited my compensation numbers to avoid down votes - yikes]

I feel like you definitely have to target particular companies and more specifically specific titles and skills to offer to do so.

My guess is trying to sell high end services as a "principal software engineer" isn't going to be enough to justify that cash comp to a lot of people hiring.

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#69
post #24

Earlier quoted context omitted.

Why are you afraid of this? My org has run a hands-on technical exam with a stack of linux admin basics (I won't enumerate them here because people do their research) but they are based on real problems we've had and the feedback is overwhelmingly "this was one of the best technical interviews I've ever had." We ask the engineer who is proctoring the interview to think about the following question: Would you want to…

Yeah, we did this at a previous employer. One example, is we had them ssh, download & extract a tarball (the Linux source, but the content doesn't matter). Sometimes, they'd gunzip to stdout. The reaction tells you a lot "lol whoopsie " followed by a quick fix: person knows what they're doing. "uh… what is going on? did I break it?" followed with general cluelessness… maybe not. That did occasionally break tmux, thou…

I never cease to be amazed at how few people really realize just how little hardware is often required for getting real work done. You'd be surprised just how much that 2GB vm with a couple cores can handle!

Re: Show HN: SadServers – Test your Linux troubleshooting skills

#70

Commenting to give this a try later, I've routinely been the person to get these kinds of gremlins escalated I've long wanted for some sort of mock, "things are broken - I want to see how you think" approach for sysad

In the "tricks of hacker news" -

     188 points by fduran 3 hours ago | unvote | flag | hide | past | favorite | 68 comments
If you click 'favorite' it will save it to your favorites list. This is a publicly visible list - yours is https://news.ycombinator.com/favorites?id=bravetraveler and mine is https://news.ycombinator.com/favorites?id=shagie which makes it easy to get a bookmark type style functionality within HN.

As I tend to favorite less often than I comment, it makes it easier to find those things I want to find again.

Post reply on HN