Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

21–30 of 424 posts

Re: How I Fired Myself

#21
News flash,

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"

If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"

They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.

As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.

Re: How I Fired Myself

#22
post #4

Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.

This. Because of this and the lack of backups.

Re: How I Fired Myself

#23
There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident.

1) A cron job for the manually task you were doing.

2) Not working directly on production.

3) Having daily backups

And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.

Re: How I Fired Myself

#24
I used to be a web freelance web developer/tech guy with one client, a designer. What made me quit was an incident where his client's Wordpress site hadn't been moved properly to the new hosting. (not by me)

The DB needed to be searched and replaced to remove all the old urls. After doing so, the wp_options cell on the production site holding much of the customizations kicked back to the defaults for the theme, the serialized data format being used was sensitive to brute DB-level changes.

I had talked to my client before about putting together a decent process including dev databases, scheduled backups, everything needed to prevent just such a screwup, but he waffled. Then blamed me when things went wrong.

I'd had enough and told him to do his own tech work, leaving him to fix his client's website himself. Being that I didn't build it, I didn't know which settings to flip back. I left freelance work and never looked back.

People and companies do this all the time, refuse to spend the time and money ensuring their systems won't break when you need them the most, then scapegoat the poor little technician when it does.

I'd like to say the answer is "don't work in such environments," but there's really no saying that it won't be this way at the next job you work, either.

I certainly wouldn't internalize any guilt being handed down, ultimately it's the founders' jobs to make sure that the proper systems are in place, after all, they have much more on the line than you do. Count it a blessing that you can just walk away and find another job.

Re: How I Fired Myself

#25
post #7
post #2

Whoever cancelled the backups was equally responsible

More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.

Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected.

What's more questionable is:

* Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter.

* Junior developers don't need an access to production machine, only sysops and maybe the technical PM.

* No backup of the production database. WTF???

If they had a hardware failure they would have been in the same shit.

Re: How I Fired Myself

#26
I really feel sorry for this guy. Accidents happen, which is why development happens in a sandboxed copy of the live system and why back ups are essential. It simple shouldn't be possible (or at least, that easy) for human error to put an entire company in jeopardy.

Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.

So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.

Re: How I Fired Myself

#28
This could've happened anyone. It's a huge shame for those in charge not for you. Any business letting such operations happen without having backups or proper user-right-management should consider why they still exists, if they really make huge amounts of many as you mentioned.

Re: How I Fired Myself

#29
post #4

Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.

We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs.

My experience is that this is the norm, not the exception.

Post reply on HN