Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limoncelli Test (Joel test for Sysadmin Teams)

everythingsysadmin.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#11

Post by antoncohen »

rhizome wrote: post #9
mechanical_fish wrote: post #5

Earlier quoted context omitted.

I kind of liked the hardware refresh policy. It's not necessarily because the hardware goes stale. It's to keep the process and the personnel from going too stale. By constantly mixing in new hardware one piece at a time, you compel code to be runnable on multiple generations of hardware at one time, avoiding flag days; you continuously shake the bugs out of new code, preventing it from growing a hardware dependency…

What you describe belongs to what is called "platform spread" and is generally something to be avoided.

What mechanical_fish describes is totally normal, and the result of normal growth and hardware refreshes. A company doesn't start with 0 servers and buy 200. They start with 5, then add 15, then 30, then 50. After three years they get to 200. Then they start replacing them in waves. New generations of hardware become more efficient, you can actually save money by replacing old hardware.

The "platform spread" is inevitable. Besides that it is impossible to physically replace all your servers at once, the bean counters also prefer it to be spread out. They don't want to replace $1,000,000 worth of servers every four years, they would rather replace $250,000 every year.

Sysadmins and devs shouldn't really care if they have five generations of hardware. There are a few things that matter so new hardware fits into the infrastructure, like having IPMI, serial ports for console redirection, enough RAM to run the apps. Try to keep as much as possible the same, like it is way better to only have 3 models of spare power supply to keep in stock vs keeping 20 models. Beyond that I don't think it matters. If every server was a different model it would suck, but having 5 generations is normal and good.

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#12

Post by mechanical_fish »

rhizome wrote: post #9
mechanical_fish wrote: post #5

Earlier quoted context omitted.

I kind of liked the hardware refresh policy. It's not necessarily because the hardware goes stale. It's to keep the process and the personnel from going too stale. By constantly mixing in new hardware one piece at a time, you compel code to be runnable on multiple generations of hardware at one time, avoiding flag days; you continuously shake the bugs out of new code, preventing it from growing a hardware dependency…

What you describe belongs to what is called "platform spread" and is generally something to be avoided.

That's interesting. Perhaps the fact that I've learned about ops entirely in the era of virtual machines running atop disposable, generic, and (in the case of AWS) entirely invisible hardware has distorted my thinking on this matter...

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#13

Post by YesThatTom »

tptacek wrote: post #3

It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test. The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole che…

You have never seen a team that does all these things? You haven't worked at Google.

PS the comments so far are excellent. I'll have a new draft soon. By the way... The hardware refresh is really about desktop/laptop management in my mind.

Oh... And I'll be teaching a half-day tutorial based on the list at Usenix LISA on Dec 5th!

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#14

Post by tptacek »

YesThatTom wrote: post #13
tptacek wrote: post #3

It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test. The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole che…

You have never seen a team that does all these things? You haven't worked at Google. PS the comments so far are excellent. I'll have a new draft soon. By the way... The hardware refresh is really about desktop/laptop management in my mind. Oh... And I'll be teaching a half-day tutorial based on the list at Usenix LISA on Dec 5th!

I have not! If this is a document that says "you'd better be doing 95% of these things, because Google does 100%", you should definitely say that more clearly!

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#15

Post by ballard »

This fails for the most important reason: backups.

Automating backups is not enough, a test restore and compare should be performed or the whole job fails.

An actual restore can fail for a variety of reasons, especially if using almost-but-not-quite-obsolete tape media.

(Think of it as continuous integration for data.)

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#16

Post by [deleted] »

tptacek wrote: post #14
YesThatTom wrote: post #13

Earlier quoted context omitted.

You have never seen a team that does all these things? You haven't worked at Google. PS the comments so far are excellent. I'll have a new draft soon. By the way... The hardware refresh is really about desktop/laptop management in my mind. Oh... And I'll be teaching a half-day tutorial based on the list at Usenix LISA on Dec 5th!

I have not! If this is a document that says "you'd better be doing 95% of these things, because Google does 100%", you should definitely say that more clearly!

[deleted]

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#17

Post by TomLimoncelli »

tptacek wrote: post #14
YesThatTom wrote: post #13

Earlier quoted context omitted.

You have never seen a team that does all these things? You haven't worked at Google. PS the comments so far are excellent. I'll have a new draft soon. By the way... The hardware refresh is really about desktop/laptop management in my mind. Oh... And I'll be teaching a half-day tutorial based on the list at Usenix LISA on Dec 5th!

I have not! If this is a document that says "you'd better be doing 95% of these things, because Google does 100%", you should definitely say that more clearly!

I dunno, that felt like bragging.

By the way... I'll be teaching a class based on The Test at Usenix LISA in December.

Also... I'm working on an edited/corrected update to The Test and I appreciate all the feedback I'm getting here on HN!

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#18

Post by TomLimoncelli »

tptacek wrote: post #3

It's not bad, but it needs editing. The Joel Test is a hugely important document (for a blog post) because it's incisive. Spolsky could have added, "does your team ban sprintf, strcpy, and strcat" and then written a graf on buffer overflows. He didn't, because that's not one of the very few questions in the Joel Test. The Joel Test isn't "if Joel Spolsky was designing a new dev team from scratch, here's his whole che…

Yes, very constructive.

A more detailed reply: 3) really? the core of devops is to be data-driven in your decisions. how can you decide if you are maintaining the right uptime if you don't measure it?

8) I agree with you, as does the text. I think you may have reversed what I wrote.

9) The good ones write so that they "think before they do". On a larger team it is important to communicate what you are about to do, or what you have done. I prefer to write mini design docs. The team I'm on does this and I like it so much I want to spread the word.

11: Again, the team I'm on does this and it works so well I want to spread the word. I see I need to expand this out to explain why, not just how.

14: I'll clarify that the point is not to make big changes on your production system. Whether it is qa+live or a zillion steps including dev, qa, UAT, pre-prod, canary and prod. As long as it isn't zero steps.

22: refresh policy: This is for PCs (non-servers). I'll clarify.

23: The last part makes your point. I'll rewrite to make it more evident

28: "Anti-malware? Really? In 2011? I'm sure you have a whole blog post". Yes, I do: http://everythingsysadmin.com/2011/04/apt.html Thanks for the reminder to add a link! (and if you are blown away that I had to list this, you can imagine my surprise about finding sites that violated this one!)

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#19

Post by TomLimoncelli »

mechanical_fish wrote: post #12
rhizome wrote: post #9

Earlier quoted context omitted.

What you describe belongs to what is called "platform spread" and is generally something to be avoided.

That's interesting. Perhaps the fact that I've learned about ops entirely in the era of virtual machines running atop disposable, generic, and (in the case of AWS) entirely invisible hardware has distorted my thinking on this matter...

"Sysadmins and devs shouldn't really care if they have five generations of hardware."

I wish it was true. sadly there are some services where scale and latency is so carefully measured that individual software releases are rejected if performance gets worse (or unacceptably worse, etc). In these situations you need to test on all hardware platforms. It is much better to have fewer platforms: Optimally: the one you are migrating off of, the one you are moving to.

For desktops... have you ever tried to maintain an Windows or Linux desktop environment with more than 4 "standard desktop configurations"? It becomes a nightmare. If you have a single "gold image" you blast to all machines it makes the task harder; if you stay with the vendor's OS and try to maintain it "forever" it is even worse.

One thing that makes virtualization a "win" is that the virtual box looks like a single hardware platform. It reduces testing, etc. However, then you still need to test the virtualization software on all hardware platforms... so you've made things easier for everyone but that team.

Re: The Limoncelli Test (Joel test for Sysadmin Teams)

#20

Post by TomLimoncelli »

antoncohen wrote: post #10
rhizome wrote: post #8

Earlier quoted context omitted.

What monitors the monitor-monitor?

The monitoring servers monitor each other (and themselves), they should be in different data centers. You can also use a third-party service to monitor parts of your infrastructure, including the monitoring server. Depending on your needs a simple service like Pingdom could be used. If you are wondering how to monitor if both data centers go don't at the same time, I'd say for most companies you don't worry about it.…

I'm not sure how email would help. I guess you mean the system would email you once a day saying that the monitoring system is working and if you don't see the email you know to check into it. The monitoring system I use has a higher SLA than 24 hours.

Usually folks divide the monitoring work among two servers and each server monitors the other. Or, you "meta monitor"... a monitoring system that just monitors the monitoring system. Then you get a third-party to monitor that. Then it is turtles all the way down.

Post reply on HN