Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I deleted data from production

blef.fr

Post reply on HN
Print view View original

31–40 of 68 posts

Re: I deleted data from production

#31

Post by winternett »

arpa wrote: post #19
psim1 wrote: post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

Mistakes happen. People forget which environment they are on. people forget where statement in their DELETE query. People misremember their cwd prior to running rm. Sometimes it's lack of experience, sometimes lack of sleep, sometimes just shit luck. Very rarely it's lack of responsibility. And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from importan…

Experienced solutions architects prevent mistakes from being costly with responsible infrastructure design, disaster recovery plans, and solid management workflows.

Companies cut corners by not hiring architects because they are not cheap, and then they cut corners on salary for roles on mission critical projects because they are also cheap, and they often pay the price anyway just later-on because failure is expensive.

Hire experienced people for the right money if it's a mission-critical system. The simple facts are all there.

Nobody successful designs a good building without a good architect, and the project never turns out well when pay isn't right.

Also important to note: The architect should not also be the one to physically build the "house", in order to maintain proper objectivity and in order to avoid conflict, even if the company insists on being cheap.

Re: I deleted data from production

#32

Post by kelnos »

tomcooks wrote: post #27
a012 wrote: post #10

The stories scream incompetent people being incompetent.

This is precisely what one thins before doing an honest mistake, then they learn they're less perfect than expected.

I agree with that in principle, but I think the things OP writes about are egregiously bad. Both in the sense of the employees in question being incredibly careless, and the company being negligent in developing process around having this sort of "god mode" access.

I've seen my share of honest mistakes (and committed some of them), but... damn.

Re: I deleted data from production

#33

Post by anyfoo »

deathanatos wrote: post #15

I've had someone do, in essence, a `terraform destroy`. I don't think we ever learned if it was that, or some other command … just that it was indeed a terraform removal. The plan was not read, of course. Instant incident. More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which w…

> I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no?

The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions?

Also, the mention of sudo makes it pretty clear that they did not do this in an interactive root shell (obtainable with sudo -s/-i or su if there is a root pw). If you do this kind of shenanigans, you better have multiple open root shells in front of you, and some statically linked binaries (busybox?) to recover from the inevitable chaos of missing binaries, shared libraries, and dynamic loaders that you'll transiently have.[1] It's not impossible to perform, but requires careful planning and a good understanding.

[1] Though if /usr was indeed copied without preserving permissions, recovering would have been very tedious.

Re: I deleted data from production

#34

Post by nicolas_t »

snicker7 wrote: post #29
sdhfjg wrote: post #18

Earlier quoted context omitted.

Read the article. We're not talking about formal verification. >It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason hdfs dfs -rm /data This is profound incompetence.

Possible explanation: lots of terminals paste on click. A single miss click can execute who know what from your clipboard.

In my experience some people keep making the same careless mistakes, the first time, you let it pass, treat it as a learning experience. The second time, you start seeing that it's always the same person doing the same mistake.

Copy pasting is not an excuse, before you run anything destructive, you double check what you're running.

Anyone who is responsible will double check before running this kind of command, and if I can't get that person off the team, I'd severely restrict his access (in general, I think most people in a team should not have access to production data).

And we do have disaster recovery plans so there's very little that could be done that would be catastrophic. But still, a lot of the disaster recovery plans call for downtime because it's not worth the cost benefit to engineer the system to be completely resilient to idiocy.

Re: I deleted data from production

#35

Post by noisy_boy »

There are three examples in the article: removing /usr (!!), running "hdfs dfs -rm /data" and "terraform destroy". Even with basic knowledge, these are clearly destructive operations - and to top it all, being run in Production. What strikes me as odd is that in all the cases, there is no change control. And by that I don't necessarily mean a full-on ServiceNow-type change process; I mean there was no "review". Even if you are a small firm, it is not difficult to ensure that you ask atleast one colleague to review any destructive step in Production prior to executing. Ignoring to do so is inviting disaster.

Re: I deleted data from production

#36

Post by macrolocal »

sdhfjg wrote: post #18
whatever1 wrote: post #14

Earlier quoted context omitted.

None of the code for consumer production is verified formally. So please spare us the bs that you make no mistakes. You are just lucky. Be humble because the complexity of modern systems is insane, there is no way you have all cases covered, if you had you would had a formal proof. We all are just doing our best to cover most of the edge cases. That is why we need to keep learning from other peoples mistakes.

Read the article. We're not talking about formal verification. >It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason hdfs dfs -rm /data This is profound incompetence.

Back in 2014, my first tech job, I wrote a clean-up script to delete HDFS artifacts listed in some text file. One day I modified the list and left a blank line at the end. :)

We had nightly back-ups though.

Re: I deleted data from production

#38

Post by wiz21c »

Talkinf of personal stories, I'd be very happy to have a tool that enforces some verification process over SQL operation ran on production database (you know, to fix that horrible data quality issue on 2 rows that screws the whole business logic in the upper layer that would take a month to fix).

Basically a tool that allow someone to run some SQL on production but, before doing it, request his credentials, then run the sql command in some "--dry-run" mode and if you modify more than 10 rows, then it stops everything. And if it goes on, record the SQL that was run somewhere so we can audit...

Re: I deleted data from production

#39

Post by bob1029 »

Instead of playing an elaborate finger pointing game, we decided to develop internal tools which put our most dangerous operational activities on rails.

It's amazing how much impact a small console application can have on preventing mistakes. All you really need to do is wrap the danger with "are you sure?" and "enter the exact commit hash..." kinds of prompts. For the super dangerous stuff you can bake in some centralization and approval loops.

Re: I deleted data from production

#40

Post by teddyh »

snicker7 wrote: post #29
sdhfjg wrote: post #18

Earlier quoted context omitted.

Read the article. We're not talking about formal verification. >It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason hdfs dfs -rm /data This is profound incompetence.

Possible explanation: lots of terminals paste on click. A single miss click can execute who know what from your clipboard.

Modern terminals marks pasted data as pasted, and similarly modern shells detect these marks and do not run the pasted data immediately, but shows it, highlighted, so that you can review it before confirming it.
Post reply on HN