Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

101–110 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#101
I'm currently working with FOIA software, and a regular user can only delete one document at a time from the information that they verify/redact before sending out. They can't even multi select! Only an admin can delete multiple documents at one time.

I'm guessing users accidentally deleted multiple documents one too many times, and now it's baked in.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#102

Any process that makes a junior directly access prod codebase/database is flawed. No matter how small of a company you are, you can set up a proper CI/CD pipeline.

90% of IT companies in Italy don't even know what a CI/CD pipeline is. That said I don't think it's something we could've integrated in our pipeline as it's an error that originated from an external service!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#103
post #70
post #66

Earlier quoted context omitted.

Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.

That was our SOP for running DELETE SQL commands on production too, a script that generates a .sql that's run manually. It saved out asses a fair amount of times

Yeah, wish I'd learned that the easy way. Fresh into one of my first jobs I was working with a vendor's custom interface to merge/purge duplicate records. It didn't have a good method of record matching on inserts from the customer web interface so a large % of records had duplicates.

Anyway, I selected what I though was a "merge all duplicates" option without previewing results. What I had actually done was "merge all selected". So, the system proceeded to merge a very large % of the database... Into One. Single. Record.

Luckily the vendor kept very good backups, and so I kept my job. Because I also luckily had a very good boss and I had already demonstrated my value in other ways, he just asked me "Well, are you going to make that mistake again?". I wisely said no, and he just smiled and said "Then I think we're done here."

I have been particularly fortunate throughout my career to have very good managers. As much as managers get a lot of flack here on HN, done well they are empowering, not a hindrance, and I attribute a lot of success in my career to them.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#104
post #50

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Owning the mistake would be fine if he did that - he did'nt. He blamed the company he was contracting for. That's a big no from me

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#105

Related: is there any HTTP API model that supports transactions with commit and rollback? Also isolation levels? Usually one wants to set_stock(get_stock() + 10) but there may be competing from various clients between both calls, resulting in races. Usual web APIs seem vulnerable to this.

Wouldn't the model be to expose an increment_stock(10) type HTTP endpoint instead, and the backend can ensure it's atomic?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#106
post #97

Earlier quoted context omitted.

This was taught to me in my first linux admin job. I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if thing…

> This stops accidentally copying a carriage return and executing the command. For a one-liner sure, but a multi line command can still be catastrophic. Showing the contents of the clipboard in the terminal itself (eg via xclip) or opening an editor and saving the contents to a file are usually better approaches. The latter let’s you craft the entire command in the editor and then run it as a script.

From [0]:

[For Bash] Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.

I have tested this on windows with a MINGW64 bash, it works similarly to how `git commit` works; by creating a new temporary file and detecting* when you close the editor.

[0] https://github.com/onceupon/Bash-Oneliner

* Actually I have no idea how this works; does bash wait for the child process to stop? does it do some posix filesystem magic to detect when the file is "free"? I can't really see other ways

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#107

The more I read about vimeo the more I wonder what's up with these guys. Only recently they made some god aweful policy changes for content creators(1), but it looks like they treat their enterprise customers just the same. Surely, there must be better alternatives for hosting videos than being at the mercy of a company who couldn't care less about big paying customers. (1) https://www.theverge.com/2022/3/18/22985820…

mux.com seems like a great alternative and is super developer focused.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#108

Earlier quoted context omitted.

Talking bad about your employer is great for finding a new job. Companies are eager to hire people who bad-talk them.

He doesn't talk bad about his employer. He talks bad about his employers client.

Tech is like any other human endeavor. People talk. People change jobs and still like the people in the place they left.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#109
post #17
post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yep, even writing a simple wildcard at command-line I will 'echo' before I 'rm'.

On computers I own, I always install "trash-cli" and i even created an alias for rm to trash. It's like rm, but it goes to the good old trash. It will not save your prod but it's pretty useful on your own computer at least.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#110

Earlier quoted context omitted.

My tip would be: read what you sign.

Just to clarify, my company is under an NDA and not personally me. It also encompasses only the actual project details so a post like this is legally compliant. (Not a lawyer, might be wrong)

In every contract I've ever signed, part of the NDA clause with my employer is that I'm also bound by NDA's my employer is bound by, so if the employer signs an NDA with a customer, I would also be bound by that. It might be worth checking your contract, otherwise having a company sign an NDA doesn't hold much weight if their staff are free to go around sharing the information themselves.
Post reply on HN