Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

251–260 of 424 posts

Re: How I Fired Myself

#251

Post by gnur »

bguthrie wrote: post #205
damm wrote: post #192

Earlier quoted context omitted.

When things like this happen, you have to realize there is more than one 'truth'. There is the truth that here is someone who truncated the users table and because of that it caused the company great harm. Here's another 'truth'. 1. The company lacked backups 2. The junior developer was on a production database. Note: I'm from the oldschool of sysops who feel that you don't give every employee the keys to your kingdo…

The problem isn't who has the keys , it's how they're used . I don't care as much if a junior developer has the prod password; I care more about building an engineering and ops team that understands that dicking around with the prod database isn't okay . Sysops and DBAs are fallible too--I've seen a lot of old school shops that relied heavily on manual migration and configuration. Automate, test, isolate and expect f…

That's why you don't give out the keys to people that know how to act responsible.

Re: How I Fired Myself

#252

Post by clavalle »

jpalacios wrote: post #135
clavalle wrote: post #85

Wow. So many mistakes. Working in production database? Bad. No backups of mission critical data? Super bad. Using a relational database as a flat data store? Super bad. Honestly...I think this company deserved what they got. Good thing the author got out of there. Hopefully in their new position they will learn better practices.

Yah this story made me cringe. What exactly do you mean by: "Using a relational database as a flat data store? Super bad." Are you referring to the users table? I am not too accustomed to using flat files, so I am curious.

Users is a bit of a core table in most applications. If they were using the relational database as it should be used there would be references to the user table elsewhere in the database.

If you tried to delete the table, it would fail stating that a deletion would violate the constraints assuming you didn't have deletions cascade automatically (which would be equally bad).

On the other hand (and it probably happened here) there will be one table with all sorts of data bolted on.

So say you want a user to have multiple pieces of armor (following the spirit of this post). You should have an armor table and a user to armor many to many table. But instead you just add an Armor column to the user record and create a new user record (with a the same username for example but with a different unique artificial key) with the new piece of armor in the armor column. Then to retrieve it you just select armor where username = whatever and iterate through the list. Adds and deletions are just as easy. So, why not? Well, duplication of data, for one thing. And no referential integrity protection for another. Delete a username and everything is deleted. Forget a where clause and you are sunk.

Re: How I Fired Myself

#253

Post by KVFinn »

>I was 22 and working at a Social Gaming startup in California.

>Part of my naive testing process involved manually clearing the RAIDS table, to then recreate it programatically.

>Listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago.

"The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".

What. The. Fuck.

The LAST person I would blame is the brand new programmer. They don't backup up their production database? If it wasn't this particular incident it would have been someone else, or a hardware failure.

Re: How I Fired Myself

#254

Post by gawker »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Agree with the point that humans are fallible. We should always have back ups. A company with 1000s of paying customers should at least take steps to protect itself from this sort of catastrophe.

Re: How I Fired Myself

#255

Post by mcritz »

blowski wrote: post #62
caseysoftware wrote: post #19

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired." Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about. And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake. When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts: 1. Doctors do make mistakes. Lots of them. All the time. 2. Ev…

> We all mess up, but only the best ones will deal with it professionally and learn from it.

This. I don't regret my life's many failures. I regret the times I've flamed out, blamed others, or ran away.

Doing things means making mistakes. You can spot a professional by how they deal with mistakes.

Re: How I Fired Myself

#256

Post by jalayr »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Completely agreed.

Re: How I Fired Myself

#257

Post by greyboy »

ferrouswheel wrote: post #200

Ah, I remember being called away from my new year holiday when an engineer dropped our entire database. This happened because they didn't realise they were connected to the production database (rather than their local dev instance). We were a business intelligence company, so that data was vital. Luckily we had a analysis cluster we could restore from, but afterwards I ensured that backups were happening... never aga…

This brings to mind on of my common responses: if it's not important enough to back up - it's not important!

It may be expensive, either in complexity, costs of storage/services, etc, but it's a necessity.

I'm curious about many of the comments in this thread - why are people logging in as table owners? It's not too difficult (for talented data-driven companies) to create roles or accounts that, while powerful, still make it difficult to drop a table and such.

Re: How I Fired Myself

#258

Post by Kiro »

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

What do you mean by "no migrations"?

Re: How I Fired Myself

#259

Post by stevekemp »

islon wrote: post #18

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a…

Turning off the wrong-server is a thing that bit me before I installed molly-guard. These days that, and similar, is a tweak I apply to all hosts I control.

(molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)

Re: How I Fired Myself

#260

Post by beilabs »

Kiro wrote: post #258
grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

What do you mean by "no migrations"?

Think of your database in terms of code version control. You want to track the changes that are made to it, adding a column could be a migration, renaming a column could be another.

A migration enables you to track the changes you made and possibly rollback to previous migrations (database states) if ever required.

Post reply on HN