Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

271–280 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#271

Post by Guvante »

hackerlight wrote: post #52
Guvante wrote: post #44

Earlier quoted context omitted.

What risk are they taking exactly? Bugs ruining the business isn't meaningful risk for the customer. It isn't like day traders are at risk of going bankrupt due to that after all. They claim liquidity is their value but given how they act they don't seem to be providing measurable liquidity, either in terms of price or volume. (Yes they increase volume by getting in the middle of trades but that isn't useful volume..…

Market risk isn't the only type of risk. Many businesses in other industries don't have market risk, that isn't abnormal. Even businesses that you would expect to be exposed to market risk aren't, since they hedge most or all of it. There's operational risk, like what brought down Knight Capital, that's a type of risk. Or the risk that you will be put out of business by competition because you were too slow to innova…

How they act is the hyper focus on first to market. HFT wants to have the first buy or sell order at price X.

Being first to market does not impact liquidity availability. After all someone else has an order at that price already.

My points about risk are beyond going long or short for a meaningful amount of time (certainly not seconds, probably not minutes) trading quickly isn't hugely impactful on end users. Thus all of the downsides of trading quickly aren't reducing risk for them.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#272

Post by wink »

mijoharas wrote: post #103
ledauphin wrote: post #18

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

At the very least have a two deploys - actually removing the old code that relies on it and then repurposing it. Giant foot gun to do it all in one especially without any automated deploys.

That assumes that you have a stable, reliable, quick process to roll out updates. Sounds like they didn't, so maybe they worked on the "oh better add this feature, it's our only chance this month" pattern.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#273

Post by alexpotato »

lukeasrodgers wrote: post #270
alexpotato wrote: post #261

Couple fun facts/stories: 1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there) You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093 2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post. There are lots of stories around this but I will share this o…

I would be interested to read these stories, but the twitter links only show a single tweet ending in the phrase "A thread." Perhaps this is a new feature of X whereby only logged-in users can see a tweet and its replies.

You are correct.

You used to be able to read any public thread without logging in.

Then it became "only logged in users" to prevent scraping for AI training data (Apparently)

Now it's "you can see the linked tweet but not the rest of the thread unless you are logged in"

Re: Knightmare: A DevOps Cautionary Tale (2014)

#274

Post by jammycakes »

MiyamotoAkira wrote: post #254
jammycakes wrote: post #241

This incident highlights a problem that is often overlooked in the debate about feature branches versus feature toggles. I've worked with both feature branches and feature toggles, and while long lived feature branches can be painful to work with what with all the conflicts, they do have the advantage that problems tend to be uncovered and resolved in development before they hit production. When feature toggles go wr…

That's interesting. Whenever I have an issue with a flag it gets picked up on dev/test/uat environments (all gets tested, especially around the code behaving the same as before with the flag off). The code change never reaches production. And if for some reason the code under the flag is wrong, and it has reached production (something unexpected, unseen), undoing the change is whatever long it takes to switch the fla…

That's a good approach if you can cleanly separate out the old code from the new code, and if you can make sure that you've got all the old functionality behind the switch. Unfortunately this can be difficult at times. Feature toggles involving UI elements, third party services or legacy code can be difficult to test automatically, for example. Another risk is accidental exposure: if a feature toggle gets switched on prematurely for whatever reason, you'll end up with broken code in production.

The cases where I've experienced problems with feature toggles have been where we thought we were swapping out all the functionality but it later turned out that due to some subtleties or nuances with the system that we weren't familiar with, we had overlooked something or other.

Feature toggles sound like a less painful way of managing changes, but you really need to have a disciplined team, a well architected codebase, comprehensive test coverage and a solid switching infrastructure to avoid getting into trouble with them. My personal recommendation is to ask the question, "What would be the damage that would happen if this feature were switched on prematurely?" and if it's not a risk you're prepared to take, that's when to move to a separate branch.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#275

Post by siftrics »

UncleMeat wrote: post #255
siftrics wrote: post #63

Earlier quoted context omitted.

When you make markets you are literally paid the spread to assume the risk of holding the position.

Okay. That's largely independent of the maximum damage caused by a bug.

No.

In this case KCG was doing the opposite of making markets --- they were taking --- they were eating the spread over and over and over again until they ran out of money.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#276

Post by siftrics »

piyh wrote: post #192
siftrics wrote: post #63

Earlier quoted context omitted.

When you make markets you are literally paid the spread to assume the risk of holding the position.

Can you break this down a bit more? I've heard about making markets in relation to FTX but didn't really get the full picture.

You can Google it or ask an LLM. It's not magic. This is common knowledge.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#277

Post by dang »

envsubst wrote: post #66
hgomersall wrote: post #41

Earlier quoted context omitted.

I'm so glad I'm not wasting my life working in finance.

[flagged]

Hey, can you please review the site guidelines and stick to them when posting here? We have to ban accounts that won't.

https://news.ycombinator.com/newsguidelines.html

Re: Knightmare: A DevOps Cautionary Tale (2014)

#278

Post by cratermoon »

hollerith wrote: post #204
cratermoon wrote: post #203

Not removing old code is akin to never throwing away food, even after it reaches its expiration date. Sure, you'll have it around next time you need it, but putting year-old yeast into your baguettes is, well, a recipe for disaster.

Also on the tiny chance you really need the old code, you can dredge it up from your version-control system.

With git, yes. When I end up working with programmers who come from using other vcs, I find they are often the ones who don't delete code, or at best comment it out. I encourage them to trust git. It takes effort (barring a system failure out of git's control) to lose code. It can take some digging to find code in the git history, but it's there. Even if you run `git gc`, the objects are still there in the repo, by default. Even in extremis, if the central repo is gone, whoever has the most recent checkout still has history.

With other, older VCS, I have less confidence.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#279

Post by brightlancer »

lukeasrodgers wrote: post #270
alexpotato wrote: post #261

Couple fun facts/stories: 1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there) You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093 2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post. There are lots of stories around this but I will share this o…

I would be interested to read these stories, but the twitter links only show a single tweet ending in the phrase "A thread." Perhaps this is a new feature of X whereby only logged-in users can see a tweet and its replies.

The Wayback Machine is pretty successful with older tweets before the login block:

https://web.archive.org/web/20200118133242/https://twitter.c...

Also, a shorthand for "give me the oldest copy of something" is

https://web.archive.org/web/0/https://example.com/

Re: Knightmare: A DevOps Cautionary Tale (2014)

#280

Post by Game_Ender »

jwestbury wrote: post #238
pxmpxm wrote: post #169

Earlier quoted context omitted.

They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45…

> as soon as you load up your trading firm with a bunch of software industry hires As a software industry hire at a hedge fund right now... I'd love to see more cross-pollination, because there are so many good things happening on both sides, and so many terrible things happening through just a sheer lack of knowledge. Change management workflows are great and should be used more in finance. But software companies sh…

> andon cord systems more often

What did you mean by this? Do you mean a way to disable systems at the flick of a switch, like dynamic feature flags?

Post reply on HN