Under NDA but I'll give rough details of what's occurring while also naming my client and disparaging them to the public. Well that's a brave move...
[deleted]
I Accidentally Deleted 7TB of Videos Before Going to Production
221–230 of 362 posts
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#222This is a great technical write up, I'd love to hear the human side of this story as well! When did you tell the higher ups that you deleted production? Was no one more senior on call to try to fix it? Did they want you to learn how to fix it? Or were you the most senior responsible for this whole area? Or did they don't know?
The first part of my write up slightly explains it but the point is that HN is the top 1%. In my current company we have 10 developers, most of them without a technical degree. They know how to do what they've been doing for the past 10 years but (as with most small companies here in Italy) people don't know what best practices are used in the industry, what a pipeline is or what a dry-run is (I learned about it today myself!).
What happened is that no one knew how to react and I was probably the best suited for it, we don't really have seniority in office.
That said when I deleted the videos I immediately told my boss. He was kind of scared but his reaction was mostly "Well, now we have to re-upload them immediately, find a way. The people that uploaded them once won't be doing it twice". I was basically left on my own to find a solution (which I luckily did).
Please note that I'm in no way blaming my company or accusing it of something, this is the standard knowledge base and way of dealing with things in many places, contrary to what working in big tech or reading HN might make you believe!
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#223Earlier quoted context omitted.
idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?
> Is there really an initial environment where rm * ; rm * ; does something different than rm * once? if * expands to the rm binary itself, maybe.
How is the system different after the first and after the second call?
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#224Earlier quoted context omitted.
there's an argument that the best people around are the people who have already (or almost) made some big mistakes. I have made a couple of huge ones - luckily I kept my job
When interviewing candidates I always enquire about their professional mistakes. Their reply often is the decider between hiring/rejecting. I want to have colleagues who admit fault, be truthful about actions which lead to the issue, and learn from it. The learning includes organisations perhaps putting additional measures in place to prevent future issues. One candidate told of a story how he was On-Call early in hi…
>When interviewing candidates I always enquire about their professional mistakes.
"You see, my biggest mistake was programming in the first place! Since then, it's just been an apology tour"
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#225What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…
Just to be fair also to some commenters, I think that the post had been edited after posting from what I remember ... so maybe the older comments are not very relevant.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#226Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…
God the feeling of having your body temp rise based purely on realizing you fucked up is so relatable.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#227Earlier quoted context omitted.
While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…
Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.
There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#228"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…
> This isn't a great article to have attached your name to A million times better than your comment.
All I did was give advice. If you don't like it it's fine.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#229Earlier quoted context omitted.
I love ZFS too but that's not really relevant to this discussion because the deleted items were on a video hosting platform and the company did already have local copies.
Yes and? Make a snapshot on live. Again, never touch data before snapshot.
This reminds of some IRC threads. You post a question and someone's answer assumes you are going to rip out and replace your existing prod setup just so you can use their pet tool.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#230Earlier quoted context omitted.
> Is there really an initial environment where rm * ; rm * ; does something different than rm * once? if * expands to the rm binary itself, maybe.
How is the system different after the first and after the second call?
If there is an rm executable in the current directory, and also one later in your PATH, the second run might use a different rm that could do whatever it wants to