71–80 of 301 posts
We didn't realize the issue until he admitted that he had run an update on the full user table (forgot a where clause) and every single email was now being funneled into his email account.
I was talking about what garbage our ethernet switches were (this was one of the earliest L3 switches-- full of blue wire bodges and buggy firmware), and how I'd already encountered a dozen different ways of crashing them.
While typing I started saying "and I bet if I send it an ICMP redirect from itself (typing) to itself (typing) it won't like that at all! (enter)" --- and over the cube partitions I hear the support desk phones start ringing. Fortunately it was a bit after the end of the day and it didn't take me long to reboot the switch.
I didn't actually expect it to kill it. I probably should have.
On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…
Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…
> "Is the light on the AIT array green or amber?"
Can you explain this? What is an AIT array?
Recovery took 30 minutes (Through Heroku support as Heroku did not allow backups-via-replication outside their own backup system), but that was a very long 30 minutes.
Second worst was a cleanup system that removed old CloudFormation stacks automatically by only retaining the latest version of a specific stack. Deployed a canary version of an edge (Nginx+Varnish) stack for internal testing. Cleanup script helpfully removed the production stack entirely.
It was 1985, I was in a VAX computer lab with about 40 other people typing on the VT100 terminals... I ran a program to compute something, and forgot that I had bumped my priority wayyy up... everything in the room stopped, even the line printer, everyone went ohhhhhh. 10 seconds later, my program finished... and everything snapped back to life. Another time, I walked into a different, bigger lab, with 100 terminals.…
Sounds more like a "I brought up production" story...
We pushed a CDN config that triggered a CDN provider bug. Took down the CDN's entire presence on one continent. Broke a whole bunch of recognizable sites for a bit.
Are you referring to the Fastly outage that happened a few weeks ago, or is this more common than I realize?
On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…
I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of god-access to the mud.
One afternoon I had a ladyfriend over that I was presumably trying to impress and she'd asked about the mud. We hopped on and I summoned into existence a harmless raggedy-ann doll. That was kind of boring so I thought it would be fun to attach an NPC script to it, -- I went through the list and saw something called a "Zombie Lord" which sounded promising. I applied it, and suddenly the doll started nattering on about the apocalypse and ran off. Turned out that it killed everyone it encountered, turned them all into zombie lords, creating an exponential wave of destruction that rapidly took over the over the whole game.
I found the mental image of some little doll running around bringing on the apocalypse to be just too funny-- until my phone started ringing. Ultimately the game state had to be reported to a backup from a day or two prior.
[I've posted a couple examples, -- I dunno which one is best, but people can vote. :)]