Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

151–160 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#151

Post by xupybd »

manicennui wrote: post #101
ledauphin wrote: post #18

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.

You have to assume people will make mistakes.

A great book on that is this https://www.thenile.co.nz/books/sidney-dekker/the-field-guid...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#152

Post by itsthecourier »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Also, api versioning. They weren't running api versioning on it, they called an old method with a new set of parameters, that shouldn't have been possible in first place

Re: Knightmare: A DevOps Cautionary Tale (2014)

#154

Post by skizm »

I feel like the first thing I would build into any automated trading system is a kill switch? then every single diff or pull request I add would have some sort of automated testing to ensure the kill switch still works. Also I'd manually flip it on/off once a day to make sure it works for real. That seems like the single most important thing to build and make sure works. Or is the system too complex for something like this and I don't understand the domain well enough?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#155

Post by SoftTalker »

movedx wrote: post #134
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts.

Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior.

At least it would have logged an error that hopefully would have been looked at.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#156

Post by TheAlchemist »

hgomersall wrote: post #41
daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

I'm so glad I'm not wasting my life working in finance.

Out of curiosity - in what domain do you work ?

I find the work in finance / tech very interesting. Societally useful ? Almost certainly not. But probably still more than most good-paying tech jobs.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#157

Post by schneems »

bluelightning2k wrote: post #135
schneems wrote: post #81

Earlier quoted context omitted.

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable a…

Great reply. Counterpoint though: that automation in and of itself is more failure area. I can imagine a similar story where the deployment pipeline incorrectly rolled back due to some change in metric format and caused the infinite loss, for example. The thing with these being a 1 in a million chance is that there's thousands of different hypothetical causes. The more parts the harder to predict an interaction and w…

Test test test. If that’s not enough, pick better tools. I’m rewriting bash scripts in rust at work because it gives me the ability to make many invalid states impossible to represent in code. Is it overkill? Maybe, but it is such a huge quality of life improvement.

Automated things can fail. Sure. But consider that playbooks are just crappy automation run by unreliable meat computers.

Also you can take an iterative approach to automation:

- manual playbook only

- automate one step of the playbook

- if it goes well, move to another. If not, run a retro to figure out out how you can improve it and try again.

Stress of failure at a job responsible for deployment architecture is manageable if you have a team and culture built around respecting that stress. There are some areas of code people are more careful around, but largely we make safety a product of our tools and processes and not some heroic “try harder not to screw up” attitude.

I find the impact of helping so many developers and their companies rewarding.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#158

Post by BoorishBears »

SoftTalker wrote: post #155
movedx wrote: post #134

Earlier quoted context omitted.

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would hav…

I think this is an example of hindsight not always being 20/20

If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened.

The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansible.com/ansible/latest/playbook_guide/playb...

The key here is CI/CD makes it so its actually less work to run that one play than it is to shoot yourself in the foot with 8 separate invocations.

Even in the fact of incompetence/laziness/oversight, the general framework makes the right choice

Re: Knightmare: A DevOps Cautionary Tale (2014)

#160

Post by TheAlchemist »

realreality wrote: post #132
eddtests wrote: post #7

Earlier quoted context omitted.

And remove easy/quick liquidity for the rest of the market? Edit: downvotes, any reason why? Or just HFT == Bad?

“The market” shouldn’t even exist.

And how would you invest in companies ?
Post reply on HN