Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#81

Post by bombcar »

takoid wrote: post #75
oceanghost wrote: post #17

Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…

Hilarious story, thanks for sharing. > "Is the light on the AIT array green or amber?" Can you explain this? What is an AIT array?

https://en.m.wikipedia.org/wiki/Advanced_Intelligent_Tape

Green = backup worked.

Re: Ask HN: Best “I brought down production” story?

#82

Post by LgWoodenBadger »

takoid wrote: post #75
oceanghost wrote: post #17

Can It be a story I was involved in but I didn't do it? I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves. Our user directories were about 100GB at the time. I had sourced this special tape drive…

Hilarious story, thanks for sharing. > "Is the light on the AIT array green or amber?" Can you explain this? What is an AIT array?

I assume the 3/4-reliable tape drive?

Re: Ask HN: Best “I brought down production” story?

#84

Post by bombcar »

haolez wrote: post #73
nullc wrote: post #68

On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…

Genuine question: what's the usefulness of Solaris' default behavior? Why kill all processes?

I assume it’s part of the shutdown process. Makes sense in a way: kill -> killall.

Re: Ask HN: Best “I brought down production” story?

#85

Post by sneak »

nullc wrote: post #80

A friend of mine ran a large and relatively popular (as in at least 30 users online at any given time ...) PvP MUD on a server of mine back in the (late?) 90s. I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of…

See also: https://en.wikipedia.org/wiki/Corrupted_Blood_incident

Re: Ask HN: Best “I brought down production” story?

#86

Post by bckr »

snow_mac wrote: post #79
nullc wrote: post #68

On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…

PHB?

Pointy-Haired Boss[]

[]https://www.wikiwand.com/en/Pointy-haired_Boss

Re: Ask HN: Best “I brought down production” story?

#87

Post by moehm »

haolez wrote: post #73
nullc wrote: post #68

On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…

Genuine question: what's the usefulness of Solaris' default behavior? Why kill all processes?

Sometimes I use kill -15 -1 on Linux to log me out, but obviously not as root.

Re: Ask HN: Best “I brought down production” story?

#88

Post by greedo »

snow_mac wrote: post #79
nullc wrote: post #68

On Linux killall lets you killall all processes matching a name. On Solaris killall kills all processes. To make matters worse, I used the command on a sever with a hung console -- so it didn't apply immediately, but later in the middle of the day the console got unhung and the main database server went down. Explaining that this was an earnest error and not something malicious to the PHBs was somewhat ... delicate.…

PHB?

Pointy Haired Bastards

Re: Ask HN: Best “I brought down production” story?

#90

Post by bombcar »

onion2k wrote: post #36
e67f70028a46fba wrote: post #9

Near miss: my first job I was working on a CRUD app for a huge bank. I was dumb and it was early in the enterprise era of software and I had built my own simple O/R tool based on codegen. Not a terrible tool all things considered and I was pretty pleased with myself. One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database. Not…

And this is why we use things like database users that don't have delete permission, and row-level security so users can delete things that don't belong to them. I have learned this from a very similar experience.

And why it’s often best to mark a record deleted and then have a reaper remove the records at a later point. But you must make sure all normal queries don’t see deleted items.
Post reply on HN