Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

341–350 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#341

Post by crispyambulance »

foota wrote: post #312
crispyambulance wrote: post #311

Earlier quoted context omitted.

> ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue... This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal th…

How many mistakes did you catch?

ONE real one and some occasional nitpicks to show that we were busy (after being caught asleep).

Was it worth it? No, I don't think so from an opportunity cost perspective-- even though we were the most junior folks there. A mind is a terrible thing to waste!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#343

Post by vsareto »

CapmCrackaWaka wrote: post #303
deltarholamda wrote: post #263

Earlier quoted context omitted.

Pffft, it's not a real panic until you weigh the pros and cons of leaving the country with nothing but the clothes on your back and becoming a illegal immigrant shepherd in a nation with too many consonants in its name. (Your description is so, so, spot on.)

The worst panic I've felt actually took me over the precipice into peaceful oblivion. I started simply saying to myself "oh well... It's just a job".

I don't think there's any public technical mistake that'll prevent you from ever getting a job in tech. Demand is just too high. Peaceful oblivion still isn't my default even though it should be.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#344

Post by gwerbret »

ricardobeat wrote: post #319
gwerbret wrote: post #290

Earlier quoted context omitted.

I had this experience when, years ago on my first day as group lead at $JOB, I was being shown a RAID 5 production server that held years of valuable, irreplaceable data (because there were no backups. Let me repeat that there were no backups). For some bizarre reason, I thought "oh cool, hot-swappable drives" and pulled one out of the rack. This naturally resulted in loud, persistent beeping from the machine, which…

Isn’t RAID 5 supposed to survive a single disk being taken out?

Yes, the array itself was fine; was just a dumb action on my part given how brittle the system was.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#345

Post by [deleted] »

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

[deleted]

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#346

Post by cat_plus_plus »

You are fine dude, you didn't delete any videos, only high availability cache of videos on a streaming site. If that was your master copy, you probably would have taken greater care, if not :-). Anyway, when working with caches that can be recreated in reasonable time, it's normal to take less care than when it comes to originals.

The only concern is Google Drive as the only backup, please make sure you have a local copy on a local RAID drive and another one regularly archived and stored in a bank locker.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#347

Post by cryptoboid »

pjerem wrote: post #109
veltas wrote: post #17

Earlier quoted context omitted.

Yep, even writing a simple wildcard at command-line I will 'echo' before I 'rm'.

On computers I own, I always install "trash-cli" and i even created an alias for rm to trash. It's like rm, but it goes to the good old trash. It will not save your prod but it's pretty useful on your own computer at least.

That's a good tip, thanks!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#348

Post by dotancohen »

jgoldshlag wrote: post #230
maweki wrote: post #223

Earlier quoted context omitted.

How is the system different after the first and after the second call?

If there is an rm executable in the current directory, and also one later in your PATH, the second run might use a different rm that could do whatever it wants to

This is actually a likely scenario, as it is common to alias rm to rm -i. Though your bash config will still run after .bashrc is nuked, some might wrap with a script instead of aliasing (e.g., to send items to Trash).

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#349

Post by justsomehnguy »

spookthesunset wrote: post #260
bzxcvbn wrote: post #99

Earlier quoted context omitted.

That's what I like about powershell. Every script can include a "SupportsShouldProcess" [1] attribute. What this means is that you can pass two new arguments to you script, which have standardized names across the whole platform: - -WhatIf to see what would happen if you run the script; - -Confirm, which asks for confirmation before any potentially destructive action. Moreover these arguments get passed down to any c…

I’m a bit confused (because I didnt read the docs)… does calling it with “—whatif” exercise the same code path as calling without, only the “do destructive stuff” automagically doesn’t do anything? Or is it a separate routine that you have to write? Cause if it is an entirely separate code path, doesn’t that introduce a case where what you say you’ll isn’t exactly what actually happens?

Well, just read the...

> because I didnt read the docs

Ouch.

> Or is it a separate routine that you have to write?

If you are writing a function or a module what would do something (eg API wrapper) then of course you need to write it yourself.

But if you are writing just a script for your mundade one-time/everyday tasks and call cmdlets what supports ShouldProcess then it works automagically. Issuing '-whatif' for the script would pass `-whatif` to any cmdlet what has 'ShouldProcess' in it's definition. Of course if someone made a cmdlet with a declared ShouldProcess but didn't write the logic to process it - you are out of luck.

But if have a spare couple of minutes check the docs in the link, it was originally a blog post by kevmarq, not a boring autodoc.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#350

Post by ivanhoe »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Beside doing this, I like to first just move files to another dir (keeping the relative path) instead of deleting them. It's basically like a DIY recycle bin.

If both paths are on the same disk moving files is a fast operation - and if you discover a screw up, you can easily undo it. On the other hand if everything still looks fine after a few days, you just `rm -rf` that folder and purge the files.

Post reply on HN