How I Fired Myself • Hacker News Board

Re: How I Fired Myself

#111

Post by desireco42 » Wed, Feb 27, 2013, 5:16 PM UTC

I was working two years ago in very successful, billion dollar startup. All developers had production access, but then, if you didn't know what you were doing, you would not be working there. Also, we didn't routinely access production and when we did, mostly for support issues on which we rotated, we did through 'rails console' environment that enforced business rules. In theory you could delete all data, but only in theory, and even then, we could restore it with minimal downtime.

I think it is obvious that CEO/CTO are the one to be held responsible here.

Re: How I Fired Myself

#112

Post by portentint » Wed, Feb 27, 2013, 5:18 PM UTC

One of my team accidentally deleted a client database, years ago.

I found out there were no backups.

The team member who deleted the db was completely stressed, of course. But my real anger was at the team's total failure to ensure we were running backups. Are you kidding me? If it wasn't someone clicking 'delete' by accident, it would've been a hard drive failure, or a power outage, or something similar.

And eventually I realized the blame lay with me. I was ultimately responsible for our backup plans, etc. - I hadn't given someone a clear direction regarding failover, redundancy, backups or security. I fixed it.

You told the story extremely well, and you handled it beautifully. There's no way anyone at that company had a right to blame you, or be angry at you, aside from the initial "WTF?!!!" reaction we'd all have.

Saying "You're lucky you're still here"...

The person who should've left was the CEO, not you.

Re: How I Fired Myself

#113

Post by cafard » Wed, Feb 27, 2013, 5:19 PM UTC

He did well to leave a company that a) had such practices in place and b) would hang out an inexperienced employee to dry like that.

Re: How I Fired Myself

#114

Post by Morendil » Wed, Feb 27, 2013, 5:21 PM UTC

So the person who made a split-second mistake while doing his all for the business was pressured into resigning - basically, got fired.

What I want to know is what happened to whoever decided that backups were a dispensable luxury? In 2010?

There's a rule that appears in Jerry Weinberg's writings - the person responsible for a X million dollar mistake (and who should be fired over such a mistake) is whoever has controlling authority over X million dollars' worth of the company's activities.

A company-killing mistake should result in the firing of the CEO, not in that of the low-level employee who committed the mistake. That's what C-level responsibility means.

(I had the same thing happen to me in the late 1990's, got fired over it. Sued my employer, who opted to settle out of court for a good sum of money to me. They knew full well they had no leg to stand on.)

Re: How I Fired Myself

#115

Post by lsc » Wed, Feb 27, 2013, 5:21 PM UTC

Piskvorrr wrote: post #99

lsc wrote: post #90

Earlier quoted context omitted.

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…

> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nu…

>> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident.

>Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.

All of the problems you describe can be solved by spare hardware and read only access to the backups. I mean, your SysAdmin needs control over the production environment, right? to do his or her job. but a sysadmin can function just fine without being able to overwrite backups. (assuming there is someone else around to admin the backup server.)

fixing my spelling now.

Yes, it's about trust. but anyone who demands absolute trust is, well, at the very least an overconfident asshole. I mean, in a properly designed backup system (and I don't have anything at all like this at the moment) I would not have write-access to the backups, and I'm majority shareholder and lead sysadmin.

That's what I'm saying... backups are primarily there when someone screwed it up... in other words, when someone was trusted (or trusted themselves) too much.

Re: How I Fired Myself

#116

Post by desireco42 » Wed, Feb 27, 2013, 5:22 PM UTC

desireco42 wrote: post #111

I was working two years ago in very successful, billion dollar startup. All developers had production access, but then, if you didn't know what you were doing, you would not be working there. Also, we didn't routinely access production and when we did, mostly for support issues on which we rotated, we did through 'rails console' environment that enforced business rules. In theory you could delete all data, but only i…

To add to this. I work again in billion dollar company (I think, they are really big0, I don't work on their main property, I have production db access. This is something senior developers definitely should have access to, but also with great privilege comes appropriate responsibility.

I routinely run reports, and sometimes I would wipe spammer out that passed our filters etc.

Re: How I Fired Myself

#117

Post by arethuza » Wed, Feb 27, 2013, 5:23 PM UTC

petriw wrote: post #70

arethuza wrote: post #49

Earlier quoted context omitted.

Out of interest, why not create a database user account that is read only and use that?

Just remember to always verify it's still read only. Or a coworker will find the login in your scripts, repurpose it, then notice they need more rights and "fix" the account for you.

"find the login in your scripts"

It's actually quite nice using a database server that doesn't require explicit credentials to be used.

Re: How I Fired Myself

#118

Post by tetsuseus » Wed, Feb 27, 2013, 5:27 PM UTC

I once fired everyone at a nonprofit foster care company with a careless query.

I cried to the sysops guy, and he gave me a full backup from 12 hours before, and before any cronjobs ran I had the database back in order.

Backups are free. It was their fault for not securing a critical asset to their business model.

Re: How I Fired Myself

#119

Post by mst » Wed, Feb 27, 2013, 5:27 PM UTC

MBCook wrote: post #79

xentronium wrote: post #16

This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible. Anecdote: I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important lat…

Ug. MyISAM. More than once (over the last few years) I've been doing some important update. I tend to do it the same way. START TRANSACTION; --run SQL-- --check results-- COMMIT; or ABORT TRANSACTION; Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup. So you always have to make a backup and check that the…

Any CREATE, ALTER or DROP on mysql does an implicit, silent COMMIT (not rollback) before running it.

PostgreSQL can do DDL inside a transaction though.

Re: How I Fired Myself

#120

Post by mst » Wed, Feb 27, 2013, 5:29 PM UTC

petriw wrote: post #70

arethuza wrote: post #49

Earlier quoted context omitted.

Out of interest, why not create a database user account that is read only and use that?

Just remember to always verify it's still read only. Or a coworker will find the login in your scripts, repurpose it, then notice they need more rights and "fix" the account for you.

I like to suffix such account names with _readonly.