Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

281–290 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#281

Post by movedx »

Lutger wrote: post #246
movedx wrote: post #134

Earlier quoted context omitted.

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

It is quite likely this would have been solved by a good automated deployment process. However, it is also quite likely that at some point a human error would creep into either the automated deployment process itself, or be 100% correctly deployed into production. At that point, if the error is as serious, Knight would still gave gone bankrupt since they had no way to mitigate these failure conditions. Being 100% fre…

All processes are fallible, but some are less so than others because we've automated a large part of them.

DevOps, and CI/CD, would have prevented Knight's issues 100%.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#282

Post by throwaway2037 »

dmurray wrote: post #233
throwaway2037 wrote: post #226

Earlier quoted context omitted.

This is a good question. In my experience, I have only see exchange trades reversed when there was a major bug in exchange software. If the bug is on the client side, tough luck. And reversing trades done on an exchange is usually a decision for the exchange regulator. It is a major event that only happens every few years -- at most -- for highly developed exchanges.

Reversing or amending a single "fat finger" trade happens all the time and the exchange generally has procedures for this that don't involve a regulator. Even in the most controversial recent example - LME cancelling a day's worth of nickel trades [0]- I understand it was their call and not any external regulator. That said, while I'd count LME as a "highly developed exchange", it's the Wild West compared to the US N…

As I understand, the LME nickel trade reversal was to prevent total meltdown due to multiple counterparties going bankrupt at the same time. To me, it was a classic case of exchange limits and poor risk control. "If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. -J. Paul Getty (of course, add some zeroes for today's world)

Also, can you explain more about what this phrase means? "it's the Wild West compared to the US NMS" Are you saying the risk limits and controls on LME are much worse that US markets?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#284

Post by nijave »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Imo it's easier to quantify the value of adding new code than removing old code (in terms of dollars). Things that are more easily to quantify the value of tend to get higher priority. Same applies to other areas like performance optimization, testing, security, patching (some of these have high level data like the cost of incidents/downtime or the cost of a breach)

Some companies have processes in place to try to balance this like 20% of time spent on tech debt reduction

Re: Knightmare: A DevOps Cautionary Tale (2014)

#285

Post by nijave »

chii wrote: post #188
kccqzy wrote: post #182

Earlier quoted context omitted.

No continuous deployment system worth its salt would allow configuration and code to be out of sync. They had a configuration change to turn on a flag that used to enable Power Peg but now enabled something else, plus a code change to reinterpret that flag differently.

the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a new flag for a new feature? Even if the technician was careful not to let prod be out of sync, it is possible that the deployment isn't instantaneous, and that the old code could've ran when the repurposed flag was turned on.

A couple solutions I've seen before

- Config gets generated to a deploy and saved as .json. Code downloads config file matching its own version or fails to start (this is a nice one since rollbacks becomes vary deterministic and code rollback is the same process as config rollback)

- Deploy code first and verify it's updated and working correctly before changing feature flag config (this one is still prone to errors without automation)

The first one can be done on Kubernetes using kustomize ConfigMap generator. The system I worked on used object storage for the config file and deploy tooling to generate it from a key value store at deploy time

Re: Knightmare: A DevOps Cautionary Tale (2014)

#286

Post by nijave »

joncrocks wrote: post #248
coldtea wrote: post #49

Earlier quoted context omitted.

> My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was d…

It depends on the business. It's not uncommon for clients to execute against different institutions' systems, and they can/would re-route flow to someone else if you're down. Think less "buying a car" and more "buying a pint of milk". If you're buying a car and the store is closed, you might come back the next day. If you're buying milk you will just go to the store down the street.

I imagine same with time based or opportunistic businesses. If the shopping channel (assuming it runs around the clock) couldn't process orders, they'd have to decide if they want to forgo selling other products to rerun the missed ones.

For certain types of entertainment like movies or sports, the sale may no longer be relevant.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#287

Post by mplewis »

Game_Ender wrote: post #280
jwestbury wrote: post #238

Earlier quoted context omitted.

> as soon as you load up your trading firm with a bunch of software industry hires As a software industry hire at a hedge fund right now... I'd love to see more cross-pollination, because there are so many good things happening on both sides, and so many terrible things happening through just a sheer lack of knowledge. Change management workflows are great and should be used more in finance. But software companies sh…

> andon cord systems more often What did you mean by this? Do you mean a way to disable systems at the flick of a switch, like dynamic feature flags?

Anyone is allowed to pull the red cord and stop the proverbial production line at any time when something is wrong and needs to be fixed.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#288

Post by daft_pink »

TheAlchemist wrote: post #156
hgomersall wrote: post #41

Earlier quoted context omitted.

I'm so glad I'm not wasting my life working in finance.

Out of curiosity - in what domain do you work ? I find the work in finance / tech very interesting. Societally useful ? Almost certainly not. But probably still more than most good-paying tech jobs.

I work in finance/tech, but my code doesn't execute million-dollar trades automatically. Cash transactions are reviewed by a human, and its mostly data analysis type work.

It's one thing to make recommendations or calculations and give the report to a human. It's another to start trading high volume in real time automatically.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#289

Post by jundl77 »

leaflets2 wrote: post #219
civilized wrote: post #150

I see a lot of criticism of the deployment, but why did the developers "repurpose an old flag" that activates 8 years dead code that you haven't deleted and that has completely unknown current functionality? That seems like the strangest decision made in this debacle.

To save time, I guess. They deleted the inactive code, so, why not, they thought. But then they forgot to deploy that change (to one server). Bugs and configuration errors will happen from time to time, and might look silly in retrospect. But the real problem was, I think, that there was no kill switch (managers and tech leads should have decided to add long ago)

I mean.. beyond that, that flag was used by a system from 8 years ago? Perhaps they simply didn't know they were re-using it.
Post reply on HN