I Accidentally Deleted 7TB of Videos Before Going to Production • Hacker News Board

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#151

Post by jraph » Thu, May 05, 2022, 1:33 PM UTC

dncornholio wrote: post #134

What is the f doing in url = f" https://api.ourservice.com/media?page{page}&step=100 ?

f for format ("formatted string").

It does the same thing as `https://api.ourservice.com/media?page${page}&step=100` [sic] in Javascript, or "https://api.ourservice.com/media?page$page&step=100" in Bash, PHP, Perl or Groovy (and other languages). It outs you into variable substitution / interpolation in the string literal.

In Python these string literals are called f-strings if you want to look it up. They are defined in PEP 498 - Literal String Interpolation [1] and available since Python 3.6.

[1] https://peps.python.org/pep-0498/

[sic] there probably would be a missing '=' in this url after "?page"

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#152

Post by b112 » Thu, May 05, 2022, 1:33 PM UTC

emerged wrote: post #143

HowardStark wrote: post #116

Earlier quoted context omitted.

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…

“I’m removing that semicolon!” (Pointing)

Parent meant this sort of pointing.

https://t.co/TjfX5K54H7

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#153

Post by hayd » Thu, May 05, 2022, 1:34 PM UTC

desarun wrote: post #141

Oh dude, we've all been there. 9 years ago I was working for a major broadcasting company in the arse end of London as a junior dev, building one of their Android apps. We'd roll features out months before & enable them with feature flags via a json file we'd manually push to a prod server at a later date. We'd just built a huge new feature letting you request content to be downloaded to your set top box remotely & i…

> Senior dev trusted me with prod deployment rights.

That part's crazy! If you think it was a firing offence wouldn't they've been fired? (I don't think it is, but obviously requires system changes/explanation.)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#154

Post by nix23 » Thu, May 05, 2022, 1:36 PM UTC

hnlmorg wrote: post #20

nix23 wrote: post #7

ZFS -> Snapshot....always!! Before touching writable-data (my personal mantra) ;)

I love ZFS too but that's not really relevant to this discussion because the deleted items were on a video hosting platform and the company did already have local copies.

Yes and? Make a snapshot on live. Again, never touch data before snapshot.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#155

Post by fifticon » Thu, May 05, 2022, 1:36 PM UTC

dncornholio wrote: post #134

What is the f doing in url = f" https://api.ourservice.com/media?page{page}&step=100 ?

if it's python, it's the formatting/interpolation string marker.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#156

Post by FunnyLookinHat » Thu, May 05, 2022, 1:37 PM UTC

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

I was actually really impressed with this individual! For someone who has less than a year of experience, they're showing quite a bit of initiative, drive, and curiosity - which really are what make or break engineers as they develop. Taking the time to do a blog post (effectively a post-mortem) and share it is even better!

And yes - I've literally done this exact same error (with TB of video data!). Spending the following week remediating all of that data loss was a great lesson in patience and attention to detail. :-)

OP: If you're ever looking for a job be sure to send me a message. Contact info in profile.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#157

Post by 692 » Thu, May 05, 2022, 1:37 PM UTC

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

there's an argument that the best people around are the people who have already (or almost) made some big mistakes.

I have made a couple of huge ones - luckily I kept my job

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#158

Post by ufo » Thu, May 05, 2022, 1:38 PM UTC

bambax wrote: post #63

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes. Also, maybe not have a delete action in the middle of a script. It's usually better to build a list of items to be deleted. In that case, two lists: items to be deleted, items to be kept. Then compare the lists: - make sure the sum of their lengths == number of total current items - make sure items_to_be_kept.length != 0 - make sure no two items appear in both lists - check some items chosen at random to see if…

What do you recommend, to not get intro trouble if there are spaces or newlines in the file names?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#159

Post by kirillzubovsky » Thu, May 05, 2022, 1:42 PM UTC

Mistakes happen. Kudos to the author on taking it as a learning opportunity. I am friends with a lot of smart devs, and many of them have dropped a production db at least once, and if not then, then accidentally emailed 10k people …etc. It happens. Work to avoid it, but plan for what to do when it inevitably happens. ¯\_(ツ)_/¯

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#160

Post by Beltiras » Thu, May 05, 2022, 1:43 PM UTC

honksillet wrote: post #115

chopin wrote: post #50

Earlier quoted context omitted.

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Agreed. But I’d also fire him from this job.

For having got into a sticky situation and out of it?