Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

141–150 of 340 posts

Re: Type in the exact number of machines to proceed

#141

Post by ahoka »

txutxu wrote: post #122
throwaway894345 wrote: post #107

Earlier quoted context omitted.

I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big…

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

"Both cloudformation stacks were identical (instance names, etc)."

This is why it's a good practice to include the environment name in the resource names when it makes sense. Even better, don't append the env name, but use it as a prefix, like ProdCustomerDb instead of CustomerDbProd. I also like to change the theme to dark mode in the production environments as most management UIs support this. One other neat trick is to color code PS1 in your Linux instances, like red for prod, green for dev.

Re: Type in the exact number of machines to proceed

#142

Post by X6S1x6Okd1st »

xaedes wrote: post #95
leetcrew wrote: post #85

Earlier quoted context omitted.

works great until some asshole puts alias harikb_script='harikb_script --do-it' in their .bashrc to eliminate this annoying step.

This would help a bit: Don't accept the "--do-it" as first parameter, make it obligatory to be the last.

If someone is a programmer and is trying to disable safety features making it slightly harder to do so doesn't really seem like the solution.

Re: Type in the exact number of machines to proceed

#143

Post by brlewis »

brundolf wrote: post #127
csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

>very experienced developers in their 40s

I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.

Re: Type in the exact number of machines to proceed

#144

Post by adiM »

roland35 wrote: post #75
Timpy wrote: post #43

Earlier quoted context omitted.

Whenever I have something in my hand that I'm about to put down for a second in the exact absent minded kind of way that would leave me searching all over the house for it 5 minutes later, I say it out loud. "Headphones on the table by front door."

Embarrassingly I once lost a hamburger while still holding it.. I had my arm propped up on a the back of the chair and it was just out of my peripheral vision. Not my smartest moment.

I lost my sunglasses when I was wearing them! We were going to a state park for a hike. It was a 2 hr ride for which I was wearing my sunglasses but forgot. As we came out of the car to start the hike, I spent 5 minutes searching for my sunglasses in my backpack until my friend asked what I was searching for .... Maybe I should be saying "sunglasses on" from now on

Re: Type in the exact number of machines to proceed

#146

Post by SkyBelow »

dheera wrote: post #40
csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

> I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. The concept makes sense, though I don't quite fully get how to translate it to other contexts besides train driving where unexpected and unpredictable events come up all the time. Let's say you're driving a car and the traffic light turns red. D…

Invert it and I think it works. Always prepare to stop at an intersection. Then point out it is green and call out you do not need to engage in stopping.

It may seem silly, but if we asked people who drive 30+ minutes every day if they have every accidentally ran a stop sign or red light, I suspect the numbers would be quite high (though they likely happen at times/places where chance of accidents are the smallest, such as empty roads late at night).

Re: Type in the exact number of machines to proceed

#147

Post by greedo »

ahoka wrote: post #141
txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

"Both cloudformation stacks were identical (instance names, etc)." This is why it's a good practice to include the environment name in the resource names when it makes sense. Even better, don't append the env name, but use it as a prefix, like ProdCustomerDb instead of CustomerDbProd. I also like to change the theme to dark mode in the production environments as most management UIs support this. One other neat trick…

I have my background colors configured for each environment so when I'm shelled into a server, I know exactly what I'm working with.

Re: Type in the exact number of machines to proceed

#148

Post by SkyBelow »

dingaling wrote: post #76
jrumbut wrote: post #72

Earlier quoted context omitted.

Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.

I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o

I think an improvement to SQL would be for insert/update/delete clauses to require a where clause and allow for something like 1=1 if you really intend to hit all rows. A safe but even more invasive would be requiring an end to the were clause as well (to prevent selecting a few but not all constraints).

Re: Type in the exact number of machines to proceed

#149

Post by greedo »

Gene_Parmesan wrote: post #130
txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massiv…

Sometimes. And sometimes they make the same mistake over and over.

We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours.

Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage.

Then a month ago, he offlines part of the SAN.

Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.

Re: Type in the exact number of machines to proceed

#150

Post by brlewis »

greedo wrote: post #147
ahoka wrote: post #141

Earlier quoted context omitted.

"Both cloudformation stacks were identical (instance names, etc)." This is why it's a good practice to include the environment name in the resource names when it makes sense. Even better, don't append the env name, but use it as a prefix, like ProdCustomerDb instead of CustomerDbProd. I also like to change the theme to dark mode in the production environments as most management UIs support this. One other neat trick…

I have my background colors configured for each environment so when I'm shelled into a server, I know exactly what I'm working with.

I'm too lazy to do this manually for each server, but I change the hostname color in my prompt based on its hash.

https://gitlab.com/brlewis/brlewis-config/-/blob/master/bash...

Post reply on HN