111–120 of 301 posts
We pushed a CDN config that triggered a CDN provider bug. Took down the CDN's entire presence on one continent. Broke a whole bunch of recognizable sites for a bit.
Are you referring to the Fastly outage that happened a few weeks ago, or is this more common than I realize?
I ran a bunch of scripts on some aux boxes, and Tokyo stopped working. Can't talk details... sadly
As an intern at a major automobile parts manufacturer, I took a hub home for a LAN party. Brought the hub back after LAN party and promptly plugged it into a network port and connected the wrong power cable. This was in the server room next to an AS/400 running production. Took a long vacation weekend as my error proceeded to shut down all production due to network issues causing the AS/400 to freak out. Cant run a c…
Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you?
How do you square those two things internally?
Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?
It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access.
Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the consequences, not the person responsible for the symptom.
I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours. https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data…
I was always worried about something like this happening so only ever provisioned (via ansible) one server at a time. When the logs showed it was fully synced, we provisioned the next node. It could take two days to add 10 nodes but I always felt much safer
I hit Shutdown, instead of restart. The server was in a colo a thousand miles away. Unfortunately at that time, a colo that didn't over overnight staff onsight (oncall yes, but not on site). Also we had no IPMI.
I had to page some poor dude at 1am to drive 30mn (each way) into the colo to push the "on" button. I felt terrible. (Small company, it was our one critical production server)
Earlier quoted context omitted.
I oversaw a Solaris machine for a very short period, and never got any reason to use it. But if you can change the signal, sending SIGHUP into everything looks like reasonable thing to do. Still, it's not something common enough to deserve it's own program.
Yep, and as weird as killall on solaris is-- the naming of killall on linux is kinda weird. It's "killbyname".
Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…
`ssh $host /bin/sh` (or another shell) should work?