Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

11–20 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#11

Post by mipmap04 »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I do this, too, but I also take a count of the expected number of items to be deleted as well. If my collection I'm iterating over doesn't have exactly that number of objects I expect, I don't proceed.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#12

Post by gilleain »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes, I find command line tools that have a "--dry-run" flag to be very helpful. If the tool (or script or whatever) is performing some destructive or expensive change, then having the ability to ask "what do you think I want to do?" is great.

It's like the difference between "do what I say" and "do what I mean"...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#13

Post by inglor_cz »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yeah, that is what I recommend too.

Instead of performing the dangerous action outright, just log a message to screen (or elsewhere) and watch what is happening.

Alternatively, or subsequently, chroot and try that stuff on some dummy data to see if it actually works.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#14

Post by ghoomketu »

The more I read about vimeo the more I wonder what's up with these guys.

Only recently they made some god aweful policy changes for content creators(1), but it looks like they treat their enterprise customers just the same.

Surely, there must be better alternatives for hosting videos than being at the mercy of a company who couldn't care less about big paying customers.

(1) https://www.theverge.com/2022/3/18/22985820/vimeo-bandwidth-...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#15

Post by dkersten »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection.

This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder.

In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#17

Post by veltas »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yep, even writing a simple wildcard at command-line I will 'echo' before I 'rm'.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#18

Post by thevinter »

legalcorrection wrote: post #3

[deleted]

You're right and I edited the company's name (might be too late but better this way). That said I'm not very happy with the experience of working for TheCompanyTM anyways so I'm in the process of switching jobs.

Thanks for the comment :)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#20

Post by hnlmorg »

nix23 wrote: post #7

ZFS -> Snapshot....always!! Before touching writable-data (my personal mantra) ;)

I love ZFS too but that's not really relevant to this discussion because the deleted items were on a video hosting platform and the company did already have local copies.
Post reply on HN