Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

131–140 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#131

Post by davbryn1 »

thevinter wrote: post #122
davbryn1 wrote: post #104

Earlier quoted context omitted.

Owning the mistake would be fine if he did that - he did'nt. He blamed the company he was contracting for. That's a big no from me

I'm sorry if it came off like that. The mistake in this case was completely mine (bad code and bad testing). The detour on the other two companies was mostly because this way of deleting/recovering stuff should've probably been avoided in the first place, other than that I'm absolutely not blaming anyone else!

Don't worry about all that - there isn't a developer worth their salt that hasn't made a mistake. But I'd consider having this blog post and HN post retracted purely for future internet checks. It isn't a reflection on you, and your honesty is fantastic. But there is a lot to be said about using a pseudonym when it comes this close to your employers

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#132

Post by dncornholio »

johnklos wrote: post #82

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that: 1) parsing a web page shouldn't be considered incredibly fraught with problems 2) that reloading web pages should be part of (1) 3) that this should ever possibly be run without validating the list of files that would be deleted So forget the specifics. Where are people lear…

Some mistakes can only be learned by making them. Sometimes you can tell someone a hundred times something, they won't learn until they experience it.

The point is not to prevent these mistakes, but to keep the consequences low.

Have backups, have version control, etc.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#133

Post by remram »

francis-io wrote: post #86
zrail wrote: post #66

Earlier quoted context omitted.

Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.

This was taught to me in my first linux admin job. I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if thing…

The 'enable-bracketed-paste' setting is an easier and more reliable way to deal with that: https://unix.stackexchange.com/a/600641/81005

It will prevent any number of newlines from running the commands if they're pasted instead of typed.

You can enable it either in .inputrc or .bashrc (with `bind 'set enable-bracketed-paste on'`)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#135

Post by photon-torpedo »

Apart from all the advice on how to do such destructive operations more safely, I think there's also a lesson to be learned about communicating more actively:

1. Vimeo responds to the original request with "will look into it", then... nothing happens? This may depend on culture, but at least from my experience in the UK, this is a very non-committal response, and if you really want them to do something, you'll need to chase them. Wait a few days and inquire if they have any estimate for when it might get done, or if they need more information. I find that the "looking into it" response is sometimes used to gauge how important the request is to you.

2. Once you go with your own solution, just drop a quick message to Vimeo: "Hey, just wanted to let you know we've found our own solution for this, and won't require your help any more. Sorry if you've already committed any resources for this task. Have a nice day, yada yada." This not just avoids what happened here, but is also a courtesy to them.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#136

Post by lpointal »

progx wrote: post #96

Now you learned what a backup is.

How can any enterprise only rely on such online services and not keep copies of their job on their own storage ?

At least store in large TB hard disks connected with a SATA adapter when needed, and put them in a case in a safe place (better: two copies, stored in two places). What is the HD + copy time price relatively to production work ?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#137

Post by [deleted] »

davbryn1 wrote: post #131
thevinter wrote: post #122

Earlier quoted context omitted.

I'm sorry if it came off like that. The mistake in this case was completely mine (bad code and bad testing). The detour on the other two companies was mostly because this way of deleting/recovering stuff should've probably been avoided in the first place, other than that I'm absolutely not blaming anyone else!

Don't worry about all that - there isn't a developer worth their salt that hasn't made a mistake. But I'd consider having this blog post and HN post retracted purely for future internet checks. It isn't a reflection on you, and your honesty is fantastic. But there is a lot to be said about using a pseudonym when it comes this close to your employers

[deleted]

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#138

Post by muglug »

The root of this particular issue was Vimeo's failure to do this migration for their customers.

Vimeo OTT has a codebase written in Rails, whereas the main PHP application is written in PHP. At the time Vimeo acquired Vimeo OTT's codebase, the Vimeo OTT codebase was small — around 10,000 lines of Ruby. Rewriting that codebase inside the Vimeo PHP application would have been a tough technical challenge for the all-Ruby team, and they'd have likely lost some people along the way and missed out on some content deals, so they decided instead to maintain two separate codebases and two separate login systems.

The video-playback and video-storage infra has since been unified, but all the business logic is still siloed.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#139

Post by BillyTheKing »

For larger 'live' production changes I've now started to rely on generative programming. I've got one script in some 'normal' programming language like javascript, or python, which in turn generates a script that contains a list of curl or other cli commands which do the actual deletion, modification, addition, etc.

This allows me to run a small sub-set of commands and test those under a live-environment before running all commands at once. In addition, this also functions as a complete log of what has been changed manually in production.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#140

Post by mmmm2 »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Another good approach is do deletions slowly. Put sleeps between each operation, and log everything. That way if you realize something is broken, you have a chance of catching it before it's too late.
Post reply on HN