Knightmare: A DevOps Cautionary Tale (2014)

hyperhello wrote: post #5

supportengineer wrote: post #2

They were missing any kind of risk mitigation steps, in their deployment practice.

There’s no money for that.

It is funny, but in one company I was working for, the more people they added the more they neglected all basics, such as backups. There were heavy processes for many things and they were followed very well, but for whatever reasons some really basic things went unnoticed for many years.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#9

Post by [deleted] » Sun, Sep 10, 2023, 8:28 PM UTC

eddtests wrote: post #7

realreality wrote: post #6

[flagged]

And remove easy/quick liquidity for the rest of the market? Edit: downvotes, any reason why? Or just HFT == Bad?

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#10

Post by hedora » Sun, Sep 10, 2023, 8:28 PM UTC

No continuous deployment system I have worked with would have blocked this particular bug.

They were in a situation where they were incrementally rolling out, but the code had a logic bug where the failure of one install within an incremental rollout step bankrupted the company.

I’d guard against this with runtime checks that the software version (e.g. git sha) matches, and also add fault injection into tests that invoke the software rollout infrastructure.