So, I was working for a direct competitor of Expedia in EU.
I was in the team handling flight research and I was adding some special rules for the markup engine.
While adding this new feature I noticed there was an instance of class getting initialized directly in a method, no dependency injection, and it was really hard to test this way since you could not fake this object.
The purpose of this object was to generate an hash for the request.
So I naturally let the IDE extract the dependency like I did many many times.
It was a really small feature and the part of the code was really old so nobody really knew all the secrets. Code review went quite ok so I just deployed it on our 65 servers.
And that's when I actually learned what a lock does on your CPU.
The flight search engine was usually hit by 350 search/s and on our dashboard I could see our CPU being constant 100%.
And that's how I learned that that class included a library that, a creation creates one write lock, and whenever you get the call to generate the hash you will use that lock.
So 350 req/s on a server were sharing only one lock.
Luckily we liked observability as a concept so production was down for less than 5 minutes and that's how I discovered why that class wasn't injected.
Eventually I had to remove that class because conceptually, without sharing the lock you would not get unique hashes but turns out that there was a workaround in place by using the name of the server and added to the hash...
Anyway that's when I caused my company to get silenced by flight metasearch engines across the world because we didn't respond for 5 minutes.
The next two hours our flight business people needed to call all our partners to unlock us.
It would have naturally happened in 6 hours but that was a lot of lost money.