Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I deleted data from production

blef.fr

Post reply on HN
Print view View original

21–30 of 68 posts

Re: I deleted data from production

#21

Post by short12 »

psim1 wrote: post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

In jest but personally I don't think you are a real programmer unless you have done something like this.

It's kind of "welcome to the club" right of passage

Re: I deleted data from production

#22

Post by lmm »

eftychis wrote: post #16
psim1 wrote: post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

I mean if people did not make mistakes we would all be writing directly in binary target or assembly. Should you ever delete folders named data -- probably not without knowing what you are deleting. Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company…

> Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company on that single hard drive not dying really? Or any system bug -- OS/firmware etc? Are you that crazy? Just short S&P long term -- it's less risky.

OP was using HDFS which is a distributed/clustered filesystem, their data was replicated across multiple machines. But "rm" still deletes it from everywhere.

Re: I deleted data from production

#23

Post by tomcooks »

psim1 wrote: post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

First take the git log out of your eye, and then you will see clearly to take the specs out of your brother's report.

Friendly advice: you don't sound like you rant, you sound like something else.

Re: I deleted data from production

#24

Post by mustardo »

My favourite was when a "helpfull" sysadmin deleted all the ActiveMQ data(base) files for a prod system (containing unprocessed queue items) I assume they got a low disk space alert and started deleting big files with no notice, recreating the messages from logs was fun :{

Re: I deleted data from production

#25

Post by eftychis »

lmm wrote: post #22
eftychis wrote: post #16

Earlier quoted context omitted.

I mean if people did not make mistakes we would all be writing directly in binary target or assembly. Should you ever delete folders named data -- probably not without knowing what you are deleting. Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company…

> Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company on that single hard drive not dying really? Or any system bug -- OS/firmware etc? Are you that crazy? Just short S&P long term -- it's less risky. OP was using HDFS which is a distributed/clustered…

Yeah I should have made yhat a bit more explicit with the single hard drive comment. Thanks @lmm!

Re: I deleted data from production

#26

Post by tomcooks »

short12 wrote: post #21
psim1 wrote: post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

In jest but personally I don't think you are a real programmer unless you have done something like this. It's kind of "welcome to the club" right of passage

Good old "human error", be glad it happens for it means you still have a job that hasn't been automated (yet).

Easy to mess things up when you routinely execute a bunch of commands everyday, many of which look similar in form but differ greatly in the pains they can create

Re: I deleted data from production

#28

Post by winternett »

thaumasiotes wrote: post #17
winternett wrote: post #12

I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior. I was supposed to just be a PM, not even supposed to be a Dev... Served me right for acting like I might have been able to fix the issue. No more vol…

> I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior. But... the database wasn't gone? If you delete a symlink, you can just put it back.

The (devops) admin deleted the link to an assets folder for all the file uploads for a pretty large web site and didn't even know how to explain what he did. We all didn't know it was a symlink that was deleted until later after the failure, but the disconnection caused the entire site's db to go corrupt... I restored the link but that didn't solve the resulting corruption, so a DB recovery/diff was in process from my desktop backup when they turned to blame on me...The DB was really FUBARED... Actually quite glad I didn't have to finish the fix on my own after all that drama TBH. :/

Re: I deleted data from production

#29

Post by snicker7 »

sdhfjg wrote: post #18
whatever1 wrote: post #14

Earlier quoted context omitted.

None of the code for consumer production is verified formally. So please spare us the bs that you make no mistakes. You are just lucky. Be humble because the complexity of modern systems is insane, there is no way you have all cases covered, if you had you would had a formal proof. We all are just doing our best to cover most of the edge cases. That is why we need to keep learning from other peoples mistakes.

Read the article. We're not talking about formal verification. >It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason hdfs dfs -rm /data This is profound incompetence.

Possible explanation: lots of terminals paste on click. A single miss click can execute who know what from your clipboard.

Re: I deleted data from production

#30

Post by kelnos »

I'm really torn on this stuff. On one hand, the big failure is a company process one: people should not have the permissions to just run "terraform destroy" and have it actually destroy a production environment (ditto for the HDFS /data deletion). Or at the very least, there should be a strong culture of never getting into the position of being able to do that without someone looking over your shoulder, double-checking every command before you run it.

But... some of this just feels like carelessness. I've certainly made my share of mistakes, but these just feel egregiously bad. This takeaway in particular is just all wrong:

> Measure the risk when you give all the permissions to one developer, one data engineer or one SRE — it means similar stories could happen

There is no need to measure that risk, because no one should have permissions to take down your production infrastructure so easily. This just seems like table stakes for running or working at a company that operates this kind of infrastructure.

I think it's three things: first, abide by the principle of least privilege, and make company-destroying permissions hard to come by; second, put safe (web, command-line, whatever) interfaces in front of common tasks that need to be done that could turn into accidental downtime; and third, when it's required that you do things outside of the safe interfaces, drill it into people that you never do them without a copilot who can check over your work, in real-time, before you do anything (and if anyone develops a reputation for being a cowboy... seriously, fire them).

Post reply on HN