Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

121–130 of 424 posts

Re: How I Fired Myself

#121

Post by Piskvorrr »

islon wrote: post #18

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a…

That's why the very first package installed after ssh-server should be molly-guard. Saved my butt a few times...

Re: How I Fired Myself

#122

Post by lsc »

Piskvorrr wrote: post #99
lsc wrote: post #90

Earlier quoted context omitted.

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…

> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nu…

>Hm. Your idea that "cloud is actually pixie dust magically solving all problems" seems to fail your very own test. Is there a way to prevent the outsourced admins from, um, destroying something when they are actively hostile? Nope, you've only added a layer of indirection.

the idea here is to make sure that the people with write-access to production don't have write-access to the backups and vis-a-vis. The point is that now two people have to screw it up before I lose data.

Outsourcing has it's place. You are an idiot if you outsource production and backups to the same people, though. This is why I think "the cloud" is a bad way of thinking about it. Linode and rackspace are completely different companies... one of them screwing it up is not going to effect the other.

Re: How I Fired Myself

#123

Post by benjaminwootton »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

I came here to type that almost exactly word for word.

This is a truly amazing story if this system was really supporting milions in revenue.

Re: How I Fired Myself

#124

Post by potatolicious »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual.

Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used to connect to database, etc...).

The correct way to respond to disasters like this is "we fucked up", not "someone fucked up".

Re: How I Fired Myself

#125

Post by benjaminwootton »

caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

I disagree with this.

In most companies there's enough process to protect you from the big screw ups.

Re: How I Fired Myself

#126

Post by hglaser »

jzelinskie wrote: post #89
tseabrooks wrote: post #29

Earlier quoted context omitted.

We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs. My experience is that this is the norm, not the exception.

As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.

The Joel Test stands the test of time: http://www.joelonsoftware.com/articles/fog0000000043.html

It doesn't cover every last thing, but a team following these practices is the kind of team you're looking for. Ask these questions at your interviews.

Re: How I Fired Myself

#127

Post by potatolicious »

jzelinskie wrote: post #89
tseabrooks wrote: post #29

Earlier quoted context omitted.

We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs. My experience is that this is the norm, not the exception.

As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.

The interview advice here is excellent. Ask questions - in the current climate they're hunting you, not the other way around.

Additionally, start networking now. Get to know ace developers in your area, and you will start hearing about top-level development shops. Go to meetups and other events where strong developers are likely to gather (or really, developers who give a shit about proper engineering) and meet people there.

It's next to impossible to know, walking into an office building, whether the company is a fucked up joke or good at what it does - people will tell you.

Re: How I Fired Myself

#128

Post by Piskvorrr »

lsc wrote: post #115
Piskvorrr wrote: post #99

Earlier quoted context omitted.

> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nu…

>> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. >Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink…

Okay, now I think I understand you, and it seems we're actually in agreement - there is still absolute power, but it's not all concentrated in one user :)

(that rouge/rogue thing is my pet peeve)

Re: How I Fired Myself

#129

Post by juan_juarez »

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.

Re: How I Fired Myself

#130

Post by greghinch »

Whoever a) gave production db access to a "junior" engineer and b) disabled backups of said database is at fault. I hope the author takes this more as a learning experience of how to (not) run a tech department than any personal fault.

Someone who has to use a GUI to manage a db at a company of that scale shouldn't have access to prod

Post reply on HN