Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

91–100 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#91
post #67

It's like the first time you run rm -rf /path/to/delete/ * And realize it is taking too long...

Can you explain? I feel like it removes / but not sure why.

   rm -rf /path/to/delete/ *
Note the space between the last / and the *

This will recursively remove the directory /path/to/delete and remove every file/directory that matches * in the current directory where 'rm' is being run.

When what was most likely meant was:

   rm -rf /path/to/delete/*
Note the lack of a space between the last / and . This will remove all files that match that reside in the /path/to/delete/ directory.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#93
post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

>literally simple prints statements

Yes, that can be a simple but powerful live on screen log. I developed a library to use an API from a SaaS vendor, in much the same way as the author. It was my first such project & I learned the hard way (wasted time, luckily no data loss or corruption) that print() was an excellent way to keep tabs on progress. On more than one occasion it saved me when the results started scrolling by and I did an oh sh*t! as I rushed to kill the job.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#94
post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

The No. 2 philosophy!

Make sure you got everything out and off before you pull up your pants, or else you better be prepared to deal with all the shit that might follow!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#95

Earlier quoted context omitted.

My tip would be: read what you sign.

Just to clarify, my company is under an NDA and not personally me. It also encompasses only the actual project details so a post like this is legally compliant. (Not a lawyer, might be wrong)

So you're not under an NDA as you wrote.

I don't know your position but I would assume a NDA is part of your freelancer or employee contract.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#97
post #66

Earlier quoted context omitted.

Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.

This was taught to me in my first linux admin job. I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if thing…

> This stops accidentally copying a carriage return and executing the command.

For a one-liner sure, but a multi line command can still be catastrophic.

Showing the contents of the clipboard in the terminal itself (eg via xclip) or opening an editor and saving the contents to a file are usually better approaches. The latter let’s you craft the entire command in the editor and then run it as a script.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#99
post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes, I find command line tools that have a "--dry-run" flag to be very helpful. If the tool (or script or whatever) is performing some destructive or expensive change, then having the ability to ask "what do you think I want to do?" is great. It's like the difference between "do what I say" and "do what I mean"...

That's what I like about powershell. Every script can include a "SupportsShouldProcess" [1] attribute. What this means is that you can pass two new arguments to you script, which have standardized names across the whole platform:

- -WhatIf to see what would happen if you run the script;

- -Confirm, which asks for confirmation before any potentially destructive action.

Moreover these arguments get passed down to any command you write in your script that support them. So you can write something like:

    [CmdletBinding(SupportsShouldProcess)]
    param ([Parameter()] [string] $FolderToBeDeleted)
    
    # I'm using bash-like aliases but these are really powershell cmdlets!
    echo "Deleting files in $FolderToBeDeleted"
    $files = @(ls $FolderToBeDeleted -rec -file)
    echo "Found $($files.Length) files"
    rm $files
If I call this script with -WhatIf, it will only display the list of files to be deleted without doing anything. If I call it with -Confirm, it will ask for confirmation before each file, with an option to abort, debug the script, or process the rest without confirming again.

I can also declare that my script is "High" impact with the "ConfirmImpact = High" switch. This will make it so that the user gets asked for confirmation without explicitly passing -Confirm. A user can set their $ConfirmPreference to High, Medium, Low, or None, to make sure they get asked for confirmation for any script that declare an impact at least as high as their preference.

[1]: https://docs.microsoft.com/en-us/powershell/scripting/learn/...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#100
A big part of the reason for the problem in this post is because Vimeo made it impossible to move videos from one Vimeo product to another Vimeo product: "There were roughly 500 videos on VimeoOTT that had to be transferred to Enterprise and Vimeo doesn't provide an easy way of doing it."

I have found working with Vimeo to be very frustrating, especially recently. They have a great video solution, especially for streaming, but they seem to put these unnecessary and frustrating roadblocks that make me constantly question my decision to use Vimeo. From in ability to move videos from one place to another, requiring complete uploads (resulting in problems like this post) to nonsensical limits and pricing, especially on their new webinar offering, which has a limit of 100 registered attendees. For anyone who has run webinars before, this makes no sense since 100 registered attendees usually means 20-30% of those people actually attend, so you're capped at 20-30 live attendees. They should price it like most event sites and charge per live attendance rather than registration.

Regardless, I've been very frustrated with Vimeo since it could be so much better if they didn't have these roadblocks in place. If they could have easily enabled moving videos from one product to another, the post (and 7TB of lost videos) would never have happened. It wasn't always this way with Vimeo, but they went IPO in May 2021 and it's no surprise they're turning the screws on their product offering and pricing now.

Post reply on HN