Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

181–190 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#181

Post by distortionfield »

amluto wrote: post #165
distortionfield wrote: post #137

Earlier quoted context omitted.

Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.

That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.

> That last nine in a trading system uptime has exponentially low value

IME that last 9 is where all the action happens

> unless you have customers who care quite a lot

All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than.

> only $100M

How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#182

Post by kccqzy »

hedora wrote: post #10

No continuous deployment system I have worked with would have blocked this particular bug. They were in a situation where they were incrementally rolling out, but the code had a logic bug where the failure of one install within an incremental rollout step bankrupted the company. I’d guard against this with runtime checks that the software version (e.g. git sha) matches, and also add fault injection into tests that in…

No continuous deployment system worth its salt would allow configuration and code to be out of sync. They had a configuration change to turn on a flag that used to enable Power Peg but now enabled something else, plus a code change to reinterpret that flag differently.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#183

Post by SamuelAdams »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever.

I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…

Re: Knightmare: A DevOps Cautionary Tale (2014)

#184

Post by chii »

SamuelAdams wrote: post #183
lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…

> All your code history is one git rebase away from being abolished forever

unless you also GC right away, it's not gone.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#186

Post by lolinder »

SamuelAdams wrote: post #183
lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…

Yes, you should definitely have branch protection turned on on main, I kind of assumed that went without saying. But to actually lose your entire git history would require both having no branch protection and having every single developer on your staff be in the regular habit of force rebasing their own branches on main. If a single developer does a double take when they're told that their branch has a different history than origin/main, then you probably didn't lose more than a week of work.

Also, it's worth noting that this dead code will almost certainly not get reused wherever you decide to store it, so it's best to keep it as far away from zombification as possible, even if it's not the safest place.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#187

Post by Terr_ »

chii wrote: post #184
SamuelAdams wrote: post #183

Earlier quoted context omitted.

Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…

> All your code history is one git rebase away from being abolished forever unless you also GC right away, it's not gone.

Yep, and also:

1. You shouldn't be allowing anybody to force-push rebased stuff onto major branches in your main repo (the one builds come from) anyway. This is especially important to support auditing and trace-ability.

2. Just because the folder is a git-repo isn't an excuse not to have it part of your regular offsite backup set.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#188

Post by chii »

kccqzy wrote: post #182
hedora wrote: post #10

No continuous deployment system I have worked with would have blocked this particular bug. They were in a situation where they were incrementally rolling out, but the code had a logic bug where the failure of one install within an incremental rollout step bankrupted the company. I’d guard against this with runtime checks that the software version (e.g. git sha) matches, and also add fault injection into tests that in…

No continuous deployment system worth its salt would allow configuration and code to be out of sync. They had a configuration change to turn on a flag that used to enable Power Peg but now enabled something else, plus a code change to reinterpret that flag differently.

the situation is caused by a confluence of multiple issues.

The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a new flag for a new feature?

Even if the technician was careful not to let prod be out of sync, it is possible that the deployment isn't instantaneous, and that the old code could've ran when the repurposed flag was turned on.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#190

Post by piyh »

movedx wrote: post #134
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

Tree, meet forest.
Post reply on HN