Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

141–150 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#141

Post by HellsMaddy »

kaashif wrote: post #112
yardstick wrote: post #86

Earlier quoted context omitted.

It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

> Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too. When I Google "zapzap machine" this comment is the only result, but after looking around on Wikipedia, I see this is a typo for "zipzap". Is this really the only time in history someone has typoed zipzap as zapzap? I guess so.

For anyone who is still confused: https://en.wikipedia.org/wiki/Credit_card_imprinter

Re: Knightmare: A DevOps Cautionary Tale (2014)

#142

Post by yardstick »

kaashif wrote: post #112
yardstick wrote: post #86

Earlier quoted context omitted.

It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

> Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too. When I Google "zapzap machine" this comment is the only result, but after looking around on Wikipedia, I see this is a typo for "zipzap". Is this really the only time in history someone has typoed zipzap as zapzap? I guess so.

Haha yeah I guess so! Last time I used one was in the previous millennium.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#143

Post by qeternity »

tomp wrote: post #14

Ah, Knight Capital. The warning story for every quant trader / engineer. This is what people don't realize when they say HFT (high frequency trading) is risk-free, leeching off people, etc. You make a million every day with very little volatility (the traditional way of quantifying "risk" in finance) but one little mistake, and you're gone. The technical term is "picking up pennies in front of a steamroller (train)".…

> Selling options is also like that.

There are plenty of bad options traders, particularly retail...but this is an oversimplification. You can buy an index fund, and it can go to zero (however unlikely). You're not guaranteed any return, whereas at least selling an option has some guaranteed fixed premium.

Professional options traders are incredibly sophisticated, and most of the tail risk is offloaded to people who are always long biased. Options as a whole massively improve price discovery in markets.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#144

Post by cwkoss »

rvz wrote: post #53

But ChatGPT would have fixed the issue faster in 45 mins than a human would. /s A high risk situation like this would make the idea of using LLMs for this as not an option; before someone puts out a 'use-case' for a LLM to fix this issue. I'm sorry to preempt the thought of this in advance, but it would not.

That's an odd nonsequitor strawman you constructed to knock down. Did someone suggest LLMs as the solution or are you just asserting superiority over an imaginary guy?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#146

Post by shiponfire »

xyst wrote: post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

[flagged]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#147

Post by bluelightning2k »

gumballindie wrote: post #22

> Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided. Would it have been avoided though? Configuration, deployment and test automation mean nothing if they don't do what they are supposed to do. Regardless of how many tests you have, if you don't test for the right stuff it's all useless.

Yes. It would have.

The specific part is configuration as code. So the config change (flag activation) and code change (flag calling) would have been synchronized.

And there wouldn't have been one server of 8 with a different build for a meaningful time and also if it did fail to deploy on that one server it would have been obvious.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#148

Post by bluelightning2k »

SoftTalker wrote: post #38
thorum wrote: post #13

Honestly seems like the market itself should have safeguards against this kind of thing.

The safeguard is "you go bankrupt if you fuck up" Imagine there was some way for a trading company to execute billions of dollars of trades and they say "ooops, sorry, that was all a mistake" can you not see how that would be abused? Now, the story also says that within a minute of the market opening, the experienced traders knew something was wrong. Do they bear any culpability for jumping on those trades, making th…

A safeguard doesn't have to be a revert.

A safeguard could be rate limiting. And or the ability to disable an API key or whatever the equivalent is.

This actually seems pretty reasonable to me.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#149

Post by squeaky-clean »

yardstick wrote: post #86
coldtea wrote: post #49

Earlier quoted context omitted.

> My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was d…

It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

Not just lost sales. I've seen a Walmart lose all ability to do credit card sales and after about 5 minutes maybe 10% of people waiting just started leaving with their groceries in their cart and a middle finger raised to the security telling them to stop.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#150

Post by civilized »

I see a lot of criticism of the deployment, but why did the developers "repurpose an old flag" that activates 8 years dead code that you haven't deleted and that has completely unknown current functionality? That seems like the strangest decision made in this debacle.
Post reply on HN