Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

241–250 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#241
This incident highlights a problem that is often overlooked in the debate about feature branches versus feature toggles.

I've worked with both feature branches and feature toggles, and while long lived feature branches can be painful to work with what with all the conflicts, they do have the advantage that problems tend to be uncovered and resolved in development before they hit production.

When feature toggles go wrong, on the other hand, they go wrong in production -- sometimes, as was the case here, with catastrophic results. I've always been nervous about the fact that feature toggles and trunk based development means merging code into main that you know for a fact to be buggy, immature, insufficiently tested and in some cases knowingly broken. If the feature toggles themselves are buggy and don't cleanly separate out your production code from your development code, you're asking for trouble.

This particular case had an additional problem: they were repurposing an existing feature toggle for something else. That's just asking for trouble.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#242

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

There are multiple real root issues here. Missing manual kill switch is also one of them.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#243

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I think the blame is not on either the devops or the developers, it is on the process. If a bug occurs than there should be atleast 5-6 different metrics / alerts that should be able to catch the bug.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#244
post #234

Earlier quoted context omitted.

> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude pattern…

> and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" I do question it, but I know the answer is hard (as you demonstrated), so I don't bother. And I'm now looking at git log docs - I fail to parse how the exclude works even looking at the docs - I don't find anything about `:(` contstruct which houses exclude keyword. But thanks for -G - that will be useful…

> I fail to parse how the exclude works even looking at the docs

Most git subcommands can take a pathspec, including git log.

The documentation you want to read is the pathspec docs, see here: https://git-scm.com/docs/gitglossary#Documentation/gitglossa...

You can see that it includes the exclude keyword, among others.

Since it applies to almost every command (from 'git add -- ' to 'git checkout -- '), it's not mentioned as clearly in individual commands.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#246
post #134

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

It is quite likely this would have been solved by a good automated deployment process. However, it is also quite likely that at some point a human error would creep into either the automated deployment process itself, or be 100% correctly deployed into production.

At that point, if the error is as serious, Knight would still gave gone bankrupt since they had no way to mitigate these failure conditions.

Being 100% free of bugs is just not a viable way to end up with safe systems.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#247

Earlier quoted context omitted.

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.

A large number of developers never master the basics, that is true. But more interestingly, absolutely zero programmers can write a good amount of code that is free of bugs.

If your road to safety is bugfree code, it will end up in an accident sooner or later, 100% guaranteed.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#248
post #49
post #42

My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Long enough means at least several hours and in this time frame we can get things back to a good state, often without much external impact. We too have manual processes in place, but for any manual process we document the rollback steps (before starting) and monitor th…

> My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was d…

It depends on the business. It's not uncommon for clients to execute against different institutions' systems, and they can/would re-route flow to someone else if you're down.

Think less "buying a car" and more "buying a pint of milk". If you're buying a car and the store is closed, you might come back the next day. If you're buying milk you will just go to the store down the street.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#249
post #234

Earlier quoted context omitted.

> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude pattern…

> and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" I do question it, but I know the answer is hard (as you demonstrated), so I don't bother. And I'm now looking at git log docs - I fail to parse how the exclude works even looking at the docs - I don't find anything about `:(` contstruct which houses exclude keyword. But thanks for -G - that will be useful…

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#250
post #234

Earlier quoted context omitted.

> and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" I do question it, but I know the answer is hard (as you demonstrated), so I don't bother. And I'm now looking at git log docs - I fail to parse how the exclude works even looking at the docs - I don't find anything about `:(` contstruct which houses exclude keyword. But thanks for -G - that will be useful…

> I fail to parse how the exclude works even looking at the docs Most git subcommands can take a pathspec, including git log. The documentation you want to read is the pathspec docs, see here: https://git-scm.com/docs/gitglossary#Documentation/gitglossa... You can see that it includes the exclude keyword, among others. Since it applies to almost every command (from 'git add -- ' to 'git checkout -- '), it's not menti…

git log and all other commands' man page should really refer back to gitglossary here then. And they should either name their argument , or specify that the given argument ( or ) is a pathspec.
Post reply on HN