Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

201–210 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#201

Post by SoftTalker »

ChrisMarshallNY wrote: post #167
siliconc0w wrote: post #161

While nice automated deployment is the wrong lesson here, it's really not anticipating backwards incompatibility and poor altering and incident training. Flags should never be reused and should be retired after they're no longer useful.

> Flags should never be reused and should be retired after they're no longer useful. That's such a "no-brainer," that I don't think it's even written down, anywhere. When I read that, I was like, "Whut?" In the Days of Yore, when we hammered programs directly into the iron as Machine Code, we would do stuff like that, but I can't even imagine doing that with any halfway modern language. They don't say, but it's proba…

Possibly Fortran.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#204

Post by hollerith »

cratermoon wrote: post #203

Not removing old code is akin to never throwing away food, even after it reaches its expiration date. Sure, you'll have it around next time you need it, but putting year-old yeast into your baguettes is, well, a recipe for disaster.

Also on the tiny chance you really need the old code, you can dredge it up from your version-control system.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#205

Post by paradox242 »

andersa wrote: post #31
ledauphin wrote: post #18

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...

This is the first thing I thought of because otherwise this story doesn't make a lot of sense.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#206

Post by atomicnumber3 »

chii wrote: post #188
kccqzy wrote: post #182

Earlier quoted context omitted.

No continuous deployment system worth its salt would allow configuration and code to be out of sync. They had a configuration change to turn on a flag that used to enable Power Peg but now enabled something else, plus a code change to reinterpret that flag differently.

the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a new flag for a new feature? Even if the technician was careful not to let prod be out of sync, it is possible that the deployment isn't instantaneous, and that the old code could've ran when the repurposed flag was turned on.

Some of the trickiness that comes from HFT and adjacent things like this is you can be working on tremendously powerful hardware but still be miserly about bits because every extra byte you have to cram into a packet is extra latency. The HFT firm I worked for would "re-use flags" in the sense that each packet literally had an 8-bit section called "flags" (and further down in the message, another 8 bit section called flags2 because of course) and each bit in there was a Boolean that could be on or off - a flag. So we weren't reusing flags as much as we were re-allocating what a high bit at that index in flags meant.

We were very conscious of this kind of error though and we managed them like Scrooge counting his farthings.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#207

Post by atomicnumber3 »

shric wrote: post #74
valdiorn wrote: post #27

Literally everyone in quant finance knows about knight capital. It even has its own phrase; "pulling a knight capital" (meaning; cutting corners on mission critical systems, even ones that can bankrupt the company in an instant, and experiencing the consequences)

Indeed, it's used in onboarding material at my employer.

Same, we knew of it as the Knightmare too.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#208

Post by gorgoiler »

Terr_ wrote: post #187
chii wrote: post #184

Earlier quoted context omitted.

> All your code history is one git rebase away from being abolished forever unless you also GC right away, it's not gone.

Yep, and also: 1. You shouldn't be allowing anybody to force-push rebased stuff onto major branches in your main repo (the one builds come from) anyway. This is especially important to support auditing and trace-ability. 2. Just because the folder is a git-repo isn't an excuse not to have it part of your regular offsite backup set.

There are reasons to want branch protection but audit isn’t one of them. An auditable system would be keeping an immutable log of the git repo actions in a separate, append-only location. It wouldn’t rely on the thing being written to all the time, by users, to never get broken. In your model, your audit history only needs one bug in the branch-protection tool for it to be destroyed.

Think of it like syslog. It’s good to keep a log of events but it’s bad to rely on the /var/log/syslog on your web server. You should be logging to a remote, append-only system.

(However I would concede that if we are talking about lawyer-proof-audit — SOC2, ISO etc — rather than actual security auditing, then branch protection is probably just fine.)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#209

Post by atomicnumber3 »

skizm wrote: post #154

I feel like the first thing I would build into any automated trading system is a kill switch? then every single diff or pull request I add would have some sort of automated testing to ensure the kill switch still works. Also I'd manually flip it on/off once a day to make sure it works for real. That seems like the single most important thing to build and make sure works. Or is the system too complex for something lik…

Most systems do. At pastjob we had a few different levels:

- halt - just stop trading

- exit-only - only exit positions (but do so according to our alphas, no hurry)

- flatten - exit in a hurry but obey certain limits (often if liquidity was thin we would just "journal" the shares - move them to a long-term-hold (meaning more than the current day) account to exit in the opening auction the next day

- market exit - get the fuck out, now, no matter what the cost.

I never saw us use that last one.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#210

Post by Narkov »

distortionfield wrote: post #181
amluto wrote: post #165

Earlier quoted context omitted.

That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.

> That last nine in a trading system uptime has exponentially low value IME that last 9 is where all the action happens > unless you have customers who care quite a lot All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than. > only $100M How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?

> How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?

If $10b is at risk, $100m is not a lot for an insurance policy.

Post reply on HN