How to lose $172,222 a second for 45 minutes • Hacker News Board

Re: How to lose $172,222 a second for 45 minutes

#241

Post by jdbernard » Wed, Oct 23, 2013, 3:47 PM UTC

twic wrote: post #191

jdbernard wrote: post #167

Earlier quoted context omitted.

If I understand correctly, they were planning to remove the Power Peg code. Do you re-test the left-overs of your refactoring: the parts you delete? The real problem here is that they were using incremental deployment and did not have a good process for ensuring the same changes were successfully made to all server.

The parts you delete aren't left over. They've been deleted. Do you test the parts of your code left behind after refactoring? Yes, absolutely . Code is either working correctly, and verified to be such by automatic testing, or it's not there. You don't leave unused code lying around to be removed later!

Right, and they were deleted. The code was not there after the refactoring. Except that because they were using an incremental deployment it was not actually removed from one of the servers. It was not really a case of untested software being run, but rather one of the servers running an older version of the software due to a deployment failure. The newer version repurposed the Power Peg flag, but since that one server was still running an older version of the code it behaved differently. It carried out the old behavior, which was not suitable for the current environment.

This is the biggest argument in my opinion against incremental deployment: it is hard to know exactly what is on any given box. Each time you push an incremental piece to a server you have effectively created a completely unique and custom version of the software on that server. Much better to package the entire solution and be able to say with certainty, "Server A has build number 123."

Re: How to lose $172,222 a second for 45 minutes

#242

Post by protomyth » Wed, Oct 23, 2013, 5:43 PM UTC

HillRat wrote: post #220

protomyth wrote: post #168

"During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review." That is just painful…

Look up the 1999 WorldCom outage due to a screwed-up load of Lucent's Jade platform upgrade. Fun times. Best I can say about that is that within a year WCOM execs had bigger problems than just pissing off CBOT....

Oh my, that sounds like a serious Career Ending Event for someone.

Re: How to lose $172,222 a second for 45 minutes

#243

Post by nasalgoat » Thu, Oct 24, 2013, 11:18 AM UTC

twic wrote: post #193

hedwall wrote: post #90

Earlier quoted context omitted.

How come? DevOps isn't a role (to begin with) and a lot of the practices documented in the text is the opposite of good DevOps practices.

AOL. True devops is about building systems that make this kind of disaster impossible. Or at least very hard. However, like agile before it, despite the fact that it really means something purposeful and rigorous, the word "devops" has become widely abused to camouflage undisciplined, thoughtless, cowboy behaviour. A handy way of telling the difference is to ask yourself "what would Devops Borat do?"; if it's somethi…

I like to think of DevOps as one of those cheap, 3-in-1 printers that you buy thinking that you'll be saving money and desk space.

Then you discover it does a mediocre job of each of those tasks as compared to a dedicated printer, scanner and fax machine. Sure, they'll take up more desk space, but you'll get higher quality results.