Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

121–130 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#121

Post by jameshart »

pc86 wrote: post #76
maweki wrote: post #69

Earlier quoted context omitted.

idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

In addition to what einsty said (which is 100% accurate), if you're deleting aged records, on any system of sufficient size objects will become aged beyond your threshold between executions.

Right. You can kind of consider the state of a filesystem on which you occasionally run rm * purges to be a system whose state is made up of ‘stuff in the filesystem’ and ‘timestamp the last purge was run’.

If you run rm * multiple times, the state of the system changes each time because that ‘timestamp’ ends up being different each time.

But if instead you run an rm on files older than a fixed timestamp, multiple times, the resulting filesystem is idempotent with respect to that operation, because the timestamp ends up set to the same value, and the filesystem in every case contains all the files added later than that timestamp.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#122

Post by thevinter »

davbryn1 wrote: post #104
chopin wrote: post #50

Earlier quoted context omitted.

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Owning the mistake would be fine if he did that - he did'nt. He blamed the company he was contracting for. That's a big no from me

I'm sorry if it came off like that. The mistake in this case was completely mine (bad code and bad testing). The detour on the other two companies was mostly because this way of deleting/recovering stuff should've probably been avoided in the first place, other than that I'm absolutely not blaming anyone else!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#123

Post by GordonS »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

It's amazing the number of times I look at some simple code and think "nah, this is so simple it doesn't need a test!", add tests anyway (because I know I should)... and immediately find the test fails because of an issue that would have been difficult to diagnose in production.

Automated tests are awesome :)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#124

Post by dredmorbius »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

mv then rm is another idiom. So long as you have the space.

For database entries, flag for deletion, then delete.

In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover.

Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specific directory, I happened to be in root.

However ...

- I'd booted a live-Linux version. (This was back when those still ran from floppy).

- I'd mounted all partitions other than the one I was performing surgery on '-ro' (read-only).

So what I bought was a reboot, and an opportunity to see what a Linux system with an active shell, but no executables, looks like.

Plan ahead. Make big changes in stages. Measure twice (or 3, or 10, or 20 times), cut once. Sit on your hands for a minute before running as root. Paste into an editor session (C-x C-e Readline command, as noted elsewhere in this thread).

Have backups.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#125

Post by GordonS »

chopin wrote: post #50
davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Aye, this is how you learn and make sure it doesn't happen again.

I did a similar thing ~20 years ago when I first started my career, accidentally deleting a production database because I thought I was working on the test database.

I owned it, learned lessons from it, and it's never happened again.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#126

Post by FriedrichN »

gilleain wrote: post #12
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Yes, I find command line tools that have a "--dry-run" flag to be very helpful. If the tool (or script or whatever) is performing some destructive or expensive change, then having the ability to ask "what do you think I want to do?" is great. It's like the difference between "do what I say" and "do what I mean"...

All my tools that have a possible destructive outcome use either a interactive stdin prompt or a --live option. I like the idea of dry running by default.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#127

Post by password4321 »

worble wrote: post #57
password4321 wrote: post #55

Earlier quoted context omitted.

SELECT COUNT(1) FROM table -- UPDATE table SET col='val' WHERE 1=1

BEGIN TRANSACTION UPDATE table SET col='val' WHERE 1=1 ROLLBACK

Definitely better, when you can afford the overhead!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#128

Post by nickkell »

davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

Better to do it before the release then afterwards. I'm assuming this way nobody noticed the issue.

Also, would you rather everyone only ever posted about all the times they were successful?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#129

Post by XorNot »

zrail wrote: post #66
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.

At the point you're doing this, you should be using a proper programming language with better defined string handling semantics though. In every place it comes up you'll have access to Python and can call the unlink command directly and much more safely - plus a debugging environment which you can actually step through if you're unsure.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#130

Post by dncornholio »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Dry run really is key here. Most automated tests wouldn't find this bug.
Post reply on HN