Partition with the mSQL database on filled up; I moved it to /tmp. On a Solaris box. Which rebooted some weeks later.
Ask HN: Best “I brought down production” story?
41–50 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#42I worked for a company back around 1999 that used MS Access to do the entire companies payroll. Including the CEO. Once I was awoken at 1am because payroll wasn’t running. In my sleepiness I accidentally manually committed a few database values (that I had never touched before) that gave the executive team 0.00 paychecks. Not a single exec including the CEO noticed for about 6 weeks.
Re: Ask HN: Best “I brought down production” story?
#43I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours.
https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on...
In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data properly which led to data being 'missing' for an hour.
In a typical Cassandra cluster when scaled up, data moves around the ring a single node at a time. When you want to add multiple nodes, this can be an extremely time and bandwidth consuming process. There's a flag called auto_bootstrap which controls this behaviour. Our understood behaviour was that it will not join the cluster until operators explicitly signal for it to so (and this is a valid scenario because as an operator, you can potentially backfill data from backups for example). Unfortunately it was completely misunderstood when we originally changed the defaults many months prior to the scale up.
Fortunately, we were able to detect data inconsistency within minutes of the original scale up and we were able to fully revert the status of the ring to it's original state within 2 hours (it took that long because we did not want to lose any new writes so we had to carefully remove nodes in the reverse order that they came in and joined the ring).
Through a mammoth effort across the engineering team across two days, we were able to reconcile the vast majority of inconsistent data through the use of audit events.
This was a mega stressful day for everyone involved. On the plus side though, I've had a few emails telling me that the blog post has saved others from making a similar mistake.
Re: Ask HN: Best “I brought down production” story?
#44Earlier quoted context omitted.
I don't get it - how would moving that binary break a running system? Is that binary somehow involved in something else beyond password changes?
/etc/passwd contains the user database on most Un*x systems. GP replaced it with the executable file, thus wiping out the system's users. Ouch.
> Ouch.
Ouch, indeed. We ended up getting lucky and found a workstation where someone had left themselves at a root prompt on another machine that had a shared NFS mount. This was before protection from this kind of attack, so we were able to create a setuid root script and run it on the main server to get root access to fix the broken passwd file.
Our next step was going to be rebooting the server. We were pretty sure that faced with a corrupt passwd file, SunOS would drop to single user mode. Never tested that theory. Glad we didn't have to, the server in question was a hack job as it was. Copied over (literally, as files) from a previous server, it wasn't even 100% in agreement with itself on its own hostname, so I always kinda wondered how it would react to any big changes.
Re: Ask HN: Best “I brought down production” story?
#45this was many years ago, as a junior dev .. management was stressing out over how to make our app faster -- long query runtimes. Naively, i pitched that we should run the queries in advance so they would be cached. simple enough. we did some dry runs. it looked good to go. we pushed to prod. its sunday night and i'm asleep, the query runner activates .. and our app proceeds to DDOS our data layer. Not only taking down our prod, but the prod of every app subscribed to the data store.
I wasn't oncall, and the oncall didnt have access to the query runner script -- it was on my laptop. So, oncall was desperately trying to fight a fire they couldn't put out while i slept like a baby .. that was a fun monday morning meeting.
Re: Ask HN: Best “I brought down production” story?
#46During my training, ~2004, I managed to kill the TCP/IP stack on an IBM mainframe running z/OS by accidentally creating a fork bomb with a perl script meant to test the performance of the newly installed BIND9.
Fortunately, it was on a testing system, SNA continued to work, and the system was due to be rebooted on the weekend anyway, so it was not that bad.
Not myself, but a few years ago, a new coworker managed to accidentally delete all user accounts from our Windows domain while trying to "clean up" the Group Policies. Our backup solution, while working, was rather crappy, so we had to restore the entire domain controller (there only was the one), which took all day, even though it was not that big. Fortunately, most users took it rather well and decided to either take the day off (it was a Friday) or tidy up their desks and sort through the papers they had lying around. A few actually thanked us for giving them the opportunity to "actually get some work done".
Re: Ask HN: Best “I brought down production” story?
#472 days before I got married, I dropped the production database by accident from a GUI tool where “right-clicking” can be destructive if you click to fast. The application scheduled radio and television commercials and within 48 US states for a large international advertising group. The bigger problem was that the DBA had only been doing incremental backups and didn’t have a full back against which to run the incremental backups. He had never created a full backup.
Fortunately, given the nature of media buys in that time, all placements were printed and faxed. My team sent me to my wedding rehearsal dinner and spent the next two days collecting printed orders and re-keying them into the system.
I am forever grateful to that team.
Re: Ask HN: Best “I brought down production” story?
#48About 5 years ago when I was just starting out I found myself designing a responsive course builder. My solution to a responsive interface at this time involved sending a very large stringified HTML file over websockets.
This wasn't a huge problem, but the configuration on Action Cable (Rails wrapper around websockets), logged the entire contents of the message to STDOUT. At a moderate scale, this combined with a memory leak bug in Docker that crashed our application every time one of our staff members tried to perform a routine action on our web app. This action resulted in a single log line of > 64kb, which Docker was unable to handle.
All of this would have been more manageable if it hadn't first surfaced while I was taxiing on a flight from Detroit to San Francisco (I was the only full time engineer). I managed to restart the application via our hosting providers mobile web interface, and frantically instructed everyone to NOT TOUCH ANYTHING until I landed.
Re: Ask HN: Best “I brought down production” story?
#49About 15 years ago, when ssh-ing into servers was quite normal.
In eterm on my gentoo linux laptop with enlightenment desktop I typed: su - shutdown -h now
Because I was tired and I wanted to go to bed. Came back after brushing my teeth. F### laptops and linux! Screen still on. The thing didn’t shutdown!
Strange thing was: in the terminal something said it got a shutdown signal.
Then I realized I shutdown a remote server for a forum with 200k members.
It was on the server of an isp employee, which happened to be member of that site. All for free, so no remote support and no kvm switches. Went to bed and took a train next day early morning to fix it.
Re: Ask HN: Best “I brought down production” story?
#50Drilling through a wall, routing a new network line; hit the power line to the server rack. There was a UPS, but it didn't like that kind of short apparently, and folded up into a sulk immediately.
Best part is that I did the wiring in that building when it was built 5 years before that; I really should have realized it was there.