161–170 of 301 posts
This one got a write up at The Register: https://www.theregister.com/2018/04/16/who_me/
The story as i told it to them is slightly different. They made some changes that I think do make it a better story, but not quite as close to the truth of the situation as I remember it.
I think that’s my biggest mistake so far.
The deployment process for a site I used to work on would entail changing the owner of a folder. You would change the owner from the web server to your user, upload files, and then change it back. sudo chown -R www-data:www-data [folder] I’d made some changes and was ready to update the owner only I was inside the folder that needed updating. In the moment I decided the correct way to refer to that folder was / I not…
I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours. https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data…
Well, one of the junior devs just so happened to be playing around with various different Solr queries to see what he could get back and somehow issued a query that caused the entire staging database to fall over. That was a fun phone call to get. It wasn’t the junior dev’s fault, of course, but it really did wonders to expose the fragility of poorly optimized/unindexed queries against the database.
My experience in general with Cassandra is that outside of a few experts working with it, it was pretty poorly understood throughout the org and no one except those select people could really do anything when it all fell over.
I'm a guy in two person startup team, mostly handle the technical stuff. Woke up to see some error on our main work machine, which also hosts some of the services, after a sudo apt update. Wasn't the first time and was usually able to fix by just googling what the error is as on SO and running it. Did it again, told me to uninstall nvidia drivers, proceeded to do so, and bricked my hard disk. Was completely gut wrenc…
Very curious what you mean by "bricked the hard disk"?
Then after a really long time an error message shows up, something like Acpi error, namespace lookup failure, Ae_not_found
Then it goes back to bios and tries again.
What I suspect happened is removing the nvidia driver led to some sort of circular dependency or lock on the system. This was when Ubuntu 20 first came out and official Ubuntu 20 cuda and nvidia drives weren't out yet so I was using ones for Ubuntu 18. Never figured it out...
So, we configured logging. Mysteriously, resets increased the next day. So we added more logging. On the third day, the whole system collapsed. Turns out the resets were coming from the super-expensive router itself because it was getting overloaded. More logging meant more load. Ooops!
After that, ops needed CEO approval to turn on any logging. Good times!
They've been good tests of our backup systems, actually. In fact, one of the incidents revealed that our on-site backups had been broken for ~a couple of weeks. (Our off-sites use a different backup system and were fine, but we restored that gap from binlog instead, as they were still present locally and it was faster than the 100Mbps upload from our off-site...)
Each one was 15-20 minutes of downtime.
Earlier quoted context omitted.
> The person responsible for the systemic failure should see the consequences You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.
Having worked in a very large, bureaucratic company I can say that I strongly suspect that just ignoring systemic failures as learning opportunities is also not a sustainable. Too many times I’ve had to yield on something and say “I guess they’ll learn when this fails” only to see them easily move on or get promoted before the failure occurs. They don’t learn their lessons. I suspect the solution is to find a way to…
Not me, but someone pinged out slack chat asking for us to "please revert". We didn't realize the issue until he admitted that he had run an update on the full user table (forgot a where clause) and every single email was now being funneled into his email account.
Earlier quoted context omitted.
I oversaw a Solaris machine for a very short period, and never got any reason to use it. But if you can change the signal, sending SIGHUP into everything looks like reasonable thing to do. Still, it's not something common enough to deserve it's own program.
Yep, and as weird as killall on solaris is-- the naming of killall on linux is kinda weird. It's "killbyname".
Nothing crazy, but something I always laugh at. I was so excited to meet a legit/professional dev team the first day of my career. I was paired with a Sr dev and sat in his cubicle so he could show me some backend update to a prod app with 20K internal users... "normally I'd run this on the dev server, but, its quick & easy so Ill just do it on prod" ...watched him crash the whole thing & struggle the rest of the day…