Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

111–120 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#111

Post by hinkley »

xyst wrote: post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

It's an entire industry built on adrenaline, bravado, and let's be honest: testosterone. How could their IT discipline be described as anything other than "YOLO"?

Trading is mostly based on a book that, like the waterfall model, was meant to be a cautionary tale on how not to do things. Liars' Poker had the exact opposite effect of Silent Spring. Imagine if Rachel Carson's book came out and people decided that a career in pesticides was more glamorous than being a doctor or a laywer, we made movies glorifying spraying pesticides everywhere and on everything, and telling anyone who thought you were crazy that they're a jealous loser and to fuck off.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#112

Post by kaashif »

yardstick wrote: post #86
coldtea wrote: post #49

Earlier quoted context omitted.

> My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was d…

It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

> Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

When I Google "zapzap machine" this comment is the only result, but after looking around on Wikipedia, I see this is a typo for "zipzap".

Is this really the only time in history someone has typoed zipzap as zapzap? I guess so.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#113

Post by hinkley »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The goal with automation is that the number of unidentified corner cases reduces over time.

A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memories from the last time they did it.

So unless there are interlocks that prevent skipping a step, it's a gamble every single time. And the effort involved in creating interlocks is a large fraction of the cost of automating.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#115

Post by CraigRo »

andersa wrote: post #31
ledauphin wrote: post #18

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...

It is very hard to change the overall size of the messages, and there's a lot of pressure to keep them short. So it could have been a bitfield or several similar things... e.g a value in a char field

Re: Knightmare: A DevOps Cautionary Tale (2014)

#116

Post by mxz3000 »

shric wrote: post #74
valdiorn wrote: post #27

Literally everyone in quant finance knows about knight capital. It even has its own phrase; "pulling a knight capital" (meaning; cutting corners on mission critical systems, even ones that can bankrupt the company in an instant, and experiencing the consequences)

Indeed, it's used in onboarding material at my employer.

Yeap, it's used as a case study for us as to the worst case scenario in trading incidents. Definitely humbling.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#117

Post by hinkley »

schneems wrote: post #81
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable a…

I wrote a tool to automate our hotfix process, and people were somewhat surprised that you could kill the process at any step and start over and it would almost always do the right thing. Like how did you expect it to work? Why replace an error prone process with an error prone and opaque one that you can't restart?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#118

Post by motoboi »

Changes we make to software and hardware infrastructure are essentially hypotheses. They're backed by evidence suggesting that these modifications will achieve our intended objectives.

What's crucial is to assess how accurately your hypothesis reflects the reality once it's been implemented. Above all, it's important to establish an instance that would definitively disprove your hypothesis - an event that wouldn't occur if your hypothesis holds true.

Harnessing this viewpoint can help you sidestep a multitude of issues.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#119

Post by hinkley »

ledauphin wrote: post #18
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

Flag recycling is a task that should be measured in months to quarters, and from what I recall of the postmortem they tried to achieve it in weeks, which is just criminally stupid.

It's this detail of the story which flips me from sympathy to schadenfreude. You dumb motherfuckers fucked around and found out.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#120

Post by hinkley »

manicennui wrote: post #101
ledauphin wrote: post #18

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.

For a group who so thoroughly despises bosses that operate on 'blame allocation', we spend a lot of time shopping around for permission to engage in reckless behavior. Most people would call that being a hypocrite.

Whereas I would call it... no, hypocrite works just fine.

Post reply on HN