Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

311–320 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#311

Post by crispyambulance »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

  > ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue...
This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal that was processed much like a silicon IC.

The process was for TWO of us to sit in front of computer and review the curves in this crazy old EDA layout tool called "L-edit" before it got sent to have the actual masks made (which were very expensive). It took HOURS to check everything.

The first hour was tolerable but then boredom started to creep in and we got sloppy. The whole reason TWO people got tasked with this was because it was thought that we would keep each other focused-- 2 pairs of eyes are better than one, right?. Instead, it just underscored the tedium of it all. One day someone walked in and found us BOTH in DEEP SLEEP in front of the monitor. Having two people didn't decrease the waste caused by mistakes, it just bored the hell out of more people.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#312

Post by foota »

crispyambulance wrote: post #311
dkersten wrote: post #15

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

> ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue... This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal th…

How many mistakes did you catch?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#313

Post by ycmjs »

gwerbret wrote: post #290
qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

I had this experience when, years ago on my first day as group lead at $JOB, I was being shown a RAID 5 production server that held years of valuable, irreplaceable data (because there were no backups. Let me repeat that there were no backups). For some bizarre reason, I thought "oh cool, hot-swappable drives" and pulled one out of the rack. This naturally resulted in loud, persistent beeping from the machine, which…

My previous boss stores company data this same way. I begged him to approve the $5 per month cost for Backblaze on the computers I used. He approved it for some, but not all (about half of the ten computers). He completely rejected the idea for the company's data. After all, it was already protected by RAID.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#315

Post by Mo3 »

ziddoap wrote: post #305
Mo3 wrote: post #288

Earlier quoted context omitted.

Seriously.. also, looking at these code snippets... If someone delivers code that looks like that, especially if intended for a production system, I'm firing immediately. It's a miracle nothing has happened sooner.

From the article: > I'm a Junior Developer with less than one year of actual experience. > The bad news is that this was on Friday, and we needed to have the videos back up at most for Tuesday morning. You say: > If someone delivers code that looks like that, especially if intended for a production system, I'm firing immediately Fire immediately? What a miserable sounding place to work.

In this case - seeing how they let them have direct access to production - I agree on the miserable sounding place to work and repeat myself -

It’s a miracle nothing happened sooner

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#316

Post by ziddoap »

Mo3 wrote: post #315
ziddoap wrote: post #305

Earlier quoted context omitted.

From the article: > I'm a Junior Developer with less than one year of actual experience. > The bad news is that this was on Friday, and we needed to have the videos back up at most for Tuesday morning. You say: > If someone delivers code that looks like that, especially if intended for a production system, I'm firing immediately Fire immediately? What a miserable sounding place to work.

In this case - seeing how they let them have direct access to production - I agree on the miserable sounding place to work and repeat myself - It’s a miracle nothing happened sooner

I was referring to your workplace.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#317

Post by ricardobeat »

marcosdumay wrote: post #249
ufo wrote: post #158

Earlier quoted context omitted.

What do you recommend, to not get intro trouble if there are spaces or newlines in the file names?

Try not to delete stuff with Bash. This is the most reliable way. Bash has a few niceties for error handling, but if you are using them, you would probably fare better in another language. If you do insist on Bash, quote everything, and use the "${var}" syntax instead of "$var". Also, make sure you handle every single possible error.

`set -e` will abort on any error, anywhere in the pipeline. It’s a must for any critical script.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#318

Post by leokennis »

RankingMember wrote: post #304

I'm impressed you went with an automated solution (PlayWright) for 500 videos after all that, considering they could be cross-loaded from Google Drive almost instantaneously. I'm glad it worked, but coding around a screw-up under the gun seems like a high-risk operation compared to spending 4 hours doing the task manually (albeit being super bored the whole time), but with the benefit of knowing it's being done corre…

Actually I was surprised reading that the person wrote a script to delete 900 videos.

If you need to do it once, it’s probably 2-3 hours of work? That is identifying a duplicate video and then clicking the button(s) to delete it once every 20 seconds.

Reminds me of https://xkcd.com/1205/

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#319

Post by ricardobeat »

gwerbret wrote: post #290
qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

I had this experience when, years ago on my first day as group lead at $JOB, I was being shown a RAID 5 production server that held years of valuable, irreplaceable data (because there were no backups. Let me repeat that there were no backups). For some bizarre reason, I thought "oh cool, hot-swappable drives" and pulled one out of the rack. This naturally resulted in loud, persistent beeping from the machine, which…

Isn’t RAID 5 supposed to survive a single disk being taken out?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#320

Post by samus »

akavel wrote: post #296
HowardStark wrote: post #116

Earlier quoted context omitted.

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…

I heard of this technique, but unfortunately I don't see how it can be easily applied in software engineering/devops. Also, I now realized that aviation checklists seem to tend to be done similarly with gestures - at least from what I saw on YouTube, not sure if that's representative or only used during education (?)

Spelling out loudly the command you are about to execute and explaining the reasoning behind it can help a lot too.
Post reply on HN