I'm guessing users accidentally deleted multiple documents one too many times, and now it's baked in.
I Accidentally Deleted 7TB of Videos Before Going to Production
101–110 of 362 posts
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#102Any process that makes a junior directly access prod codebase/database is flawed. No matter how small of a company you are, you can set up a proper CI/CD pipeline.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#103Earlier quoted context omitted.
Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.
That was our SOP for running DELETE SQL commands on production too, a script that generates a .sql that's run manually. It saved out asses a fair amount of times
Anyway, I selected what I though was a "merge all duplicates" option without previewing results. What I had actually done was "merge all selected". So, the system proceeded to merge a very large % of the database... Into One. Single. Record.
Luckily the vendor kept very good backups, and so I kept my job. Because I also luckily had a very good boss and I had already demonstrated my value in other ways, he just asked me "Well, are you going to make that mistake again?". I wisely said no, and he just smiled and said "Then I think we're done here."
I have been particularly fortunate throughout my career to have very good managers. As much as managers get a lot of flack here on HN, done well they are empowering, not a hindrance, and I attribute a lot of success in my career to them.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#104"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…
I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#105Related: is there any HTTP API model that supports transactions with commit and rollback? Also isolation levels? Usually one wants to set_stock(get_stock() + 10) but there may be competing from various clients between both calls, resulting in races. Usual web APIs seem vulnerable to this.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#106Earlier quoted context omitted.
This was taught to me in my first linux admin job. I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if thing…
> This stops accidentally copying a carriage return and executing the command. For a one-liner sure, but a multi line command can still be catastrophic. Showing the contents of the clipboard in the terminal itself (eg via xclip) or opening an editor and saving the contents to a file are usually better approaches. The latter let’s you craft the entire command in the editor and then run it as a script.
[For Bash] Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
I have tested this on windows with a MINGW64 bash, it works similarly to how `git commit` works; by creating a new temporary file and detecting* when you close the editor.
[0] https://github.com/onceupon/Bash-Oneliner
* Actually I have no idea how this works; does bash wait for the child process to stop? does it do some posix filesystem magic to detect when the file is "free"? I can't really see other ways
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#107The more I read about vimeo the more I wonder what's up with these guys. Only recently they made some god aweful policy changes for content creators(1), but it looks like they treat their enterprise customers just the same. Surely, there must be better alternatives for hosting videos than being at the mercy of a company who couldn't care less about big paying customers. (1) https://www.theverge.com/2022/3/18/22985820…
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#108Earlier quoted context omitted.
Talking bad about your employer is great for finding a new job. Companies are eager to hire people who bad-talk them.
He doesn't talk bad about his employer. He talks bad about his employers client.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#109> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…
Yep, even writing a simple wildcard at command-line I will 'echo' before I 'rm'.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#110Earlier quoted context omitted.
My tip would be: read what you sign.
Just to clarify, my company is under an NDA and not personally me. It also encompasses only the actual project details so a post like this is legally compliant. (Not a lawyer, might be wrong)