The Limoncelli Test (Joel test for Sysadmin Teams)
everythingsysadmin.com
The Limoncelli Test (Joel test for Sysadmin Teams)
1–10 of 21 posts
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#2I give it a C, maybe a B if you're entry level.
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#3It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test.
The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole checklist". But this sysadmin test seems that way.
So in that spirit, here's my first wave of things I think you should cut:
(3) Plenty of very excellent ops team don't keep internal team metrics (other than availability stats). It's also too fuzzy.
(8) Prioritizing features over stability actually contradicts Joel on Software ("some bugs aren't worth fixing", to paraphrase). Saying that you prioritize one over the other is also a platitude. Axe this.
(9) Virtually every great dev team uses source control, has bug tracking, &c. Not every great ops team has "design docs" for every (or even any) project they undertake
(11) Similarly, an "opsdoc" for every service (the mini website with "how to rebuild this") is a nice-to-have, not a must have. How I know that is, I've never once met an ops team that actually has this.
(14) Dev/QA/Prod environments: I don't think you can axe this, but I think you got too aspirational, on two axes: first, most teams don't have dev AND QA AND prod (though every good ops team has at least a prod and a "something else" environment) and secondly, there are services that don't need this much rigor.
(22) Refresh policy for hardware? If it ain't broke, &c. Why does a good sysadmin team refresh hardware just for the hell of it? I remember when network admins used to be proud of keeping highly utilized networks running on the old ugly Cisco AGS+ boxes and made fun of the kids who bragged about their 7500s. This is too fuzzy to be part of a "test".
(23) Why do I care if servers stay up when one hard drive dies, as long as my service stays up even when a whole rack catches fire?
(28) Anti-malware? Really? In 2011? I'm sure you have a whole blog post to write about this, but if you have to justify it, maybe leave it out of your "test".
A suggestion for your document that also adds another acid test to things you should keep on the list: what's an ops team that everyone knows that kicks ass as a result of doing all these things? When Joel Spolsky wrote The Joel Test, he got to use Microsoft as a "12 out of 12" case. Who does all these things? Amazon? (Did I miss that in your document? I'm tipsy, sorry).
Hope that's constructive.
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#4I give it a C, maybe a B if you're entry level.
Fine, how's this for question 18: how do you know when your notification system goes down?
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#5It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test. The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole che…
I kind of liked the hardware refresh policy. It's not necessarily because the hardware goes stale. It's to keep the process and the personnel from going too stale.
By constantly mixing in new hardware one piece at a time, you compel code to be runnable on multiple generations of hardware at one time, avoiding flag days; you continuously shake the bugs out of new code, preventing it from growing a hardware dependency in year N that only gets discovered in year N+2; you periodically drill the team (especially the newer folks) in the procedure for bringing up new boxes but also accomplish real work (gradual upgrade of the server farm) in the process; you'll end up running hardware with a continuous range of model numbers and batches, perhaps mitigating against flaws that strike entire batches at once; when disaster strikes and you have to replace a box ASAP, odds are better that you've set up a similar box in recent memory and know exactly what to get, how to set it up, and what any pitfalls might be - and if there are pitfalls, you discovered them during working hours on spare hardware, rolled back, and spent a few weeks fixing them instead of discovering them at 5 AM on a Saturday morning and having to fix them on the fly.
(Of course, my devops team does everything in AWS, so what do I know about managing hardware?)
I agree that this checklist is way too long to vie with the Joel Test, though.
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#6This is good list. I think a key part is "The score doesn't matter as much as attitude." Not every company will have everything on the list, you can't expect small startups to have all this. But if they balk at the ideas it shows that there is a problem.
It turns out that some companies have a management team that is against automation, written policies, and fixing security and stability issues. Here are questions I wish I asked in job interviews:
4. Do you have a "policy and procedure" wiki?
8. In your bugs/tickets, does stability have a higher priority than new features?
16. Do you use configuration management tools like cfengine/puppet/chef?
20. Is OS installation automated?
28. Do desktops/laptops/servers run self-updating, silent, anti-malware software?
29. Do you have a written security policy?
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#7I give it a C, maybe a B if you're entry level.
Fine, how's this for question 18: how do you know when your notification system goes down?
> 18. Do automated processes that generate email only do so when they have something to say?
> for question 18: how do you know when your notification system goes down?
You monitor your monitoring system. I think 18 is important, noise from automated processes will hide real problems. I worked with a manager who would consistently write cron jobs that run as root (17), that would send out useless emails every day (18). One of the cron jobs sent 500KB - 10MB of text every day, no one will read 10MB of text, so if there is a an error no one will see it. Write your scripts correctly, use --quiet flags and redirect stdout to /dev/null.
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#8Earlier quoted context omitted.
Fine, how's this for question 18: how do you know when your notification system goes down?
> 18. Do automated processes that generate email only do so when they have something to say? > for question 18: how do you know when your notification system goes down? You monitor your monitoring system. I think 18 is important, noise from automated processes will hide real problems. I worked with a manager who would consistently write cron jobs that run as root (17), that would send out useless emails every day (18…
What monitors the monitor-monitor?
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#9It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test. The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole che…
I kind of liked the hardware refresh policy. It's not necessarily because the hardware goes stale. It's to keep the process and the personnel from going too stale. By constantly mixing in new hardware one piece at a time, you compel code to be runnable on multiple generations of hardware at one time, avoiding flag days; you continuously shake the bugs out of new code, preventing it from growing a hardware dependency…
What you describe belongs to what is called "platform spread" and is generally something to be avoided.
Re: The Limoncelli Test (Joel test for Sysadmin Teams)
#10Earlier quoted context omitted.
> 18. Do automated processes that generate email only do so when they have something to say? > for question 18: how do you know when your notification system goes down? You monitor your monitoring system. I think 18 is important, noise from automated processes will hide real problems. I worked with a manager who would consistently write cron jobs that run as root (17), that would send out useless emails every day (18…
What monitors the monitor-monitor?
The monitoring servers monitor each other (and themselves), they should be in different data centers. You can also use a third-party service to monitor parts of your infrastructure, including the monitoring server. Depending on your needs a simple service like Pingdom could be used.
If you are wondering how to monitor if both data centers go don't at the same time, I'd say for most companies you don't worry about it. 1) The odds are extremely low. 2) You will notice if two DCs go down. 3) That nightly email that says "I'm up" isn't going to help here. 4) Even the free version of Pingdom will alert you when your whole datacenter is down.
There are all sorts of other things to consider with redundant monitoring, but that's the job of a sysadmin -- identifying failure points, assessing risk, etc.