Live data from Hacker News

The day I locked everyone out of the company intranet

dancowell.com

41–50 of 130 posts

Re: The day I locked everyone out of the company intranet

#41
post #26

Earlier quoted context omitted.

he only recreated the ones not saved by the nightly backup, meaning he used the previous nightly backup and recreated from invoices the last ~23 hours.

No, I did the entire table. That helped me find corner cases and errors in my logic having such a bigger range of source data.

Wow,

I'd still not touch previous data.

I'd use it just for ensuring correctness, but I'd not touch it

Re: The day I locked everyone out of the company intranet

#42
post #23

Always do a select with your criteria before doing a Delete or update. Don’t ask me how I learned this.

Once I wanted to do `rm -fr *~` to delete backup files, but the `~` key didn't register... Now I have learnt to instinctively stop before doing anything destructive and double-check and double-check again! This also applies to SQL `DELETE` and `UPDATE`! I know that `-r` was not neccessary but hey that was a biiiig mistake of mine!

If you ever type really dangerous commands, it is good practice to prefix them with a space (or whatever your favorite shell convention is) to make sure they not saved in your history.

One of my "oopsies" involved aggressive and usage and a previous `rm -rf *` running in /home...

Re: The day I locked everyone out of the company intranet

#43
post #26

Earlier quoted context omitted.

he only recreated the ones not saved by the nightly backup, meaning he used the previous nightly backup and recreated from invoices the last ~23 hours.

No, I did the entire table. That helped me find corner cases and errors in my logic having such a bigger range of source data.

Maybe I don't fully comprehend what you're saying... but is it even possible to do this?

I mean what if at some point particular entries were manually tweaked and the database was updated to fix an error in an invoice or something. And then you recreated data from what you assume is 100% reliable data.

I'm happy to try to understand if you don't mind simplifying the explanation.

Re: The day I locked everyone out of the company intranet

#44
I did something worst many years ago: I was working for a regional ISP and during a major incident, I had to reroute traffic through a different path. Under big pressure, I did the infamous Cisco mistake "switchport trunk allowed vlan 50" instead that "switchport trunk allowed add vlan 50" and I locked out myself and all the customer from our broadband customers. We had to call a DC technician and ask him to share a console through a local console server.

Lesson learned: even if you are under big pressure take your time to plan and review the modification 15 minutes can save hours.

Re: The day I locked everyone out of the company intranet

#45
post #23

Always do a select with your criteria before doing a Delete or update. Don’t ask me how I learned this.

Once I wanted to do `rm -fr *~` to delete backup files, but the `~` key didn't register... Now I have learnt to instinctively stop before doing anything destructive and double-check and double-check again! This also applies to SQL `DELETE` and `UPDATE`! I know that `-r` was not neccessary but hey that was a biiiig mistake of mine!

For entirely critical systems, I by now rather generate a reviewable script and run that. Something along the lines of:

    find /backups/ -mtime '-30' -printf 'rm -f %p\n' > very_scary_deletes.sh
This gives you a static script with a bunch of rm's. You can read that, check it, give it to people to validate and when you eventually run it, it deletes exactly those files.

Re: The day I locked everyone out of the company intranet

#47
post #42
post #23

Earlier quoted context omitted.

Once I wanted to do `rm -fr *~` to delete backup files, but the `~` key didn't register... Now I have learnt to instinctively stop before doing anything destructive and double-check and double-check again! This also applies to SQL `DELETE` and `UPDATE`! I know that `-r` was not neccessary but hey that was a biiiig mistake of mine!

If you ever type really dangerous commands, it is good practice to prefix them with a space (or whatever your favorite shell convention is) to make sure they not saved in your history. One of my "oopsies" involved aggressive and usage and a previous `rm -rf *` running in /home...

I didn't know about the space thing!

Though for me, I think the greater risk would be from not having a record of what I'd run.

Re: The day I locked everyone out of the company intranet

#48

Always do a select with your criteria before doing a Delete or update. Don’t ask me how I learned this.

At a previous place I worked, if they were working on the cli (eg in psql or similar) they'd always use these two steps, either of which would provide adequate protection:

1. Start a transaction before even thinking of writing the delete/update/etc (BEGIN; ...)

2. Always write the WHERE query out first, THEN go back to the start of the line and fill out the DELETE/UPDATE/etc.

It worked well, and it's a habit I've since tried to keep on doing myself as well.

Re: The day I locked everyone out of the company intranet

#49
I did worse.

We had a redis with sessions. Early on, someone decided every write to redis should also cause a write to S3 as backup. My first task was to get rid of this 4-digits a month extra cost in PUT requests. I decided to instead write all changed session objects into a set keyed by half-hour timestamps and then write only those sessions every 30 minutes. Unfortunately initially I used a KEYS to find the set corresponding to my half-hour stamp, not having read up exactly on what it does. It's not exactly advisable to do on a redis with a million or so objects. A later version of the archiver wrote the last emptied set to a stable key instead and then checked the set keys between then and now instead...

Re: The day I locked everyone out of the company intranet

#50
post #39

Earlier quoted context omitted.

> can you rollback a drop table >_ The first thing I did after the recovery marathon was to alias rm so that it instead works by moving stuff to /tmp

In the late 90’s I wanted to try this Linux thing, so I followed a tutorial. First step: fdisk Yes, that was my Windows partition going bye-bye.

So... success? ;)
Post reply on HN