Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

71–80 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#71

Post by pc86 »

bambax wrote: post #63
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes. Also, maybe not have a delete action in the middle of a script. It's usually better to build a list of items to be deleted. In that case, two lists: items to be deleted, items to be kept. Then compare the lists: - make sure the sum of their lengths == number of total current items - make sure items_to_be_kept.length != 0 - make sure no two items appear in both lists - check some items chosen at random to see if…

I've had good success with this approach, have two distinct scripts generate the two lists, then in addition to your items here also checking that every item appears in one of the lists.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#72

Post by pc86 »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I just want to say as someone currently working on a script to delete approximately 3.2TB of a ~4TB production database, this subthread is pure gold.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#73

Post by einsty »

maweki wrote: post #69
maccard wrote: post #61

Earlier quoted context omitted.

Deleting * may not be though. Your selection needs to be idempotent.

idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

In the case of any live system, i would say yes. Additional, and different, files could have appeared on the file system in between the times of each rm *.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#75

Post by SnowHill9902 »

Related: is there any HTTP API model that supports transactions with commit and rollback? Also isolation levels? Usually one wants to set_stock(get_stock() + 10) but there may be competing from various clients between both calls, resulting in races. Usual web APIs seem vulnerable to this.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#76

Post by pc86 »

maweki wrote: post #69
maccard wrote: post #61

Earlier quoted context omitted.

Deleting * may not be though. Your selection needs to be idempotent.

idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

In addition to what einsty said (which is 100% accurate), if you're deleting aged records, on any system of sufficient size objects will become aged beyond your threshold between executions.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#79

Post by p0d »

For many years I have had a private blog. I like to write but realised 99% of us are not interesting to read. This is a young guy processing his thoughts. Not "teaching" the rest of us as he frames it. This should have stayed in-house and personal. The company can then decide which clients, authorities to contact if necessary. There is a book in all of us as they say. For most of us it should stay there.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#80

Post by kwertyoowiyop »

davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

Will every developer who has never checked in bad code on Friday, or accidentally deleted the wrong data, please raise their hand?

‘Judgment comes from experience, and experience comes from poor judgment.’

:-)

Post reply on HN