Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

281–290 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#281

Post by bzxcvbn »

spookthesunset wrote: post #260
bzxcvbn wrote: post #99

Earlier quoted context omitted.

That's what I like about powershell. Every script can include a "SupportsShouldProcess" [1] attribute. What this means is that you can pass two new arguments to you script, which have standardized names across the whole platform: - -WhatIf to see what would happen if you run the script; - -Confirm, which asks for confirmation before any potentially destructive action. Moreover these arguments get passed down to any c…

I’m a bit confused (because I didnt read the docs)… does calling it with “—whatif” exercise the same code path as calling without, only the “do destructive stuff” automagically doesn’t do anything? Or is it a separate routine that you have to write? Cause if it is an entirely separate code path, doesn’t that introduce a case where what you say you’ll isn’t exactly what actually happens?

It's the first option. And yes, sometimes you have to be careful if you want to implement SupportsShouldProcess correctly, it's not something you can add willy-nilly. For example, if you create a folder, you can't `cd` there in -WhatIf mode.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#282

Post by dclowd9901 »

His solution reminds me of how I used Cypress to generate test accounts on our local admin dashboard for Cypress tests, since our api was inadequate (it didn't do the billing signoff required to create accounts that last longer than a month... don't ask...).

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#283

Post by ufmace »

dncornholio wrote: post #132
johnklos wrote: post #82

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that: 1) parsing a web page shouldn't be considered incredibly fraught with problems 2) that reloading web pages should be part of (1) 3) that this should ever possibly be run without validating the list of files that would be deleted So forget the specifics. Where are people lear…

Some mistakes can only be learned by making them. Sometimes you can tell someone a hundred times something, they won't learn until they experience it. The point is not to prevent these mistakes, but to keep the consequences low. Have backups, have version control, etc.

True, and worth remembering why. Most of us are constantly getting warned about the dire potential consequences of huge numbers of things, most of which are either massively unlikely to ever happen or not actually that bad, or both. It's very difficult to tell which of the things we get warned about are actually high risk until something bites us.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#284

Post by wonderwonder »

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

lol, its amazing how fast the blood leaves your face when your mind transitions from "cool that worked well" to "Oh no, what have I done?"

That backups comment sounds very familiar.

I accidentally deleted a clients products table from the production database in my early years as a solo dev. There was only a production database. Luckily I had written a feature to export the products to an excel sheet a while before and happened to have an excel copy from the prior day. I managed to build an export to ingest the excel and repopulate the table in record speed while waiting for my phone to ring and the client to be furious. Luckily they never found out.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#286

Post by donalhunt »

fwiw I would probably have turned to rclone.org for this. It doesn't have support for vimeo out of the box but the Vimeo API seems sane enough that it would be trivial to implement uploads quickly.

Previously used rclone for doing massive transfers between cloud providers using "cheap" on-demand servers which provide unlimited data transfer (the public clouds make this very expensive).

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#288

Post by Mo3 »

johnklos wrote: post #82

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that: 1) parsing a web page shouldn't be considered incredibly fraught with problems 2) that reloading web pages should be part of (1) 3) that this should ever possibly be run without validating the list of files that would be deleted So forget the specifics. Where are people lear…

Seriously.. also, looking at these code snippets...

If someone delivers code that looks like that, especially if intended for a production system, I'm firing immediately.

It's a miracle nothing has happened sooner.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#289

Post by water8 »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

It never hurts to ask for another set of eyes to review. At the least if something goes awry, the blame isn't solely on you.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#290

Post by gwerbret »

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

I had this experience when, years ago on my first day as group lead at $JOB, I was being shown a RAID 5 production server that held years of valuable, irreplaceable data (because there were no backups. Let me repeat that there were no backups). For some bizarre reason, I thought "oh cool, hot-swappable drives" and pulled one out of the rack. This naturally resulted in loud, persistent beeping from the machine, which everyone ignored on the assumption that the fellow who was just hired as the group lead knew what the f he was doing.

While I didn't know what I was doing, I did manage to get the beeping to stop, and had to come in at 5 a.m. the next day to restripe the drive I'd yanked out.

Did I mention there were no backups? When I was a little bit more seasoned on the job, I raised a polite but persistent issue with management of the need for durable backups. Although I kept at it for months, they thought about it, talked about it, and ultimately did nothing. A few months after I left, the entire array failed. Since the group's work relied on the irreplaceable data, all work ground to a halt for the several months it took for an off-site company to recover the data.

Post reply on HN