Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

241–250 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#241

Post by sergiotapia »

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

I lost 1hr and 30 minutes of a Slack like app (chat messages). Luckily at the time we were pretty small so not much data was lost but holy shit did that make me almost throw up.

Thank God my automatic backups were so close to the mistake I made and I didn't lose 24 hours.

Haven't made a mistake like that since and I don't destroy DB records like that anymore.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#244

Post by kortex »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

This is why I like to always write any sort of user-script batch-job tools (backfills, purges, scrapers) with a "porcelain and plumbing" approach: The first step generates a fully declarative manifest of files/uris/commands (usually just json) and the second step actually executes them. I've used a --dry-run flag to just output the manifest, but I just read some folks use a --live-run flag to enable, with dry-run being the default, and I like that much better so I'll be using that going forward.

This pattern has the added benefit that it makes it really easy to write unit tests, which is something often sorely lacking in these sorts of batch scripts. It also makes full automation down the line a breeze, since you have nice shearing layers between your components.

http://www.laputan.org/mud/mud.html#ShearingLayers

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#246

Post by furyofantares »

Great post and great attitude.

I think I would reflect on why this is a script to begin with. It's run once and with only 500 items could be done manually, though 500 is certainly a bit much.

But it's not a massive time saver; the point of the script should be almost entirely to increase accuracy. I think I would write one script to generate the list of videos to delete; that's the part that's actually difficult, and a human can then verify the list. I would probably just delete them by hand after that, but if I really wanted a script for that part too, it would be a separate script that uses a list that has been vetted by a human even if initially created by the first script.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#247

Post by kevinmgranger »

ufo wrote: post #158
bambax wrote: post #63

Earlier quoted context omitted.

Yes. Also, maybe not have a delete action in the middle of a script. It's usually better to build a list of items to be deleted. In that case, two lists: items to be deleted, items to be kept. Then compare the lists: - make sure the sum of their lengths == number of total current items - make sure items_to_be_kept.length != 0 - make sure no two items appear in both lists - check some items chosen at random to see if…

What do you recommend, to not get intro trouble if there are spaces or newlines in the file names?

Don't use a shell script.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#248

Post by marcosdumay »

dredmorbius wrote: post #124
dkersten wrote: post #15

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

mv then rm is another idiom. So long as you have the space. For database entries, flag for deletion, then delete. In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover. Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specif…

You mean cp then rm?

And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail.

Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable.

Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#249

Post by marcosdumay »

ufo wrote: post #158
bambax wrote: post #63

Earlier quoted context omitted.

Yes. Also, maybe not have a delete action in the middle of a script. It's usually better to build a list of items to be deleted. In that case, two lists: items to be deleted, items to be kept. Then compare the lists: - make sure the sum of their lengths == number of total current items - make sure items_to_be_kept.length != 0 - make sure no two items appear in both lists - check some items chosen at random to see if…

What do you recommend, to not get intro trouble if there are spaces or newlines in the file names?

Try not to delete stuff with Bash.

This is the most reliable way. Bash has a few niceties for error handling, but if you are using them, you would probably fare better in another language.

If you do insist on Bash, quote everything, and use the "${var}" syntax instead of "$var". Also, make sure you handle every single possible error.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#250

Post by ectopod »

bambax wrote: post #63
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes. Also, maybe not have a delete action in the middle of a script. It's usually better to build a list of items to be deleted. In that case, two lists: items to be deleted, items to be kept. Then compare the lists: - make sure the sum of their lengths == number of total current items - make sure items_to_be_kept.length != 0 - make sure no two items appear in both lists - check some items chosen at random to see if…

This. The original approach can fail horribly if there's a problem on the server when you run the script for real. Your code can be perfect but that's no guarantee the server will always return what it ought to.
Post reply on HN