Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

291–300 of 424 posts

Re: How I Fired Myself

#291

Post by chris_mahan »

Retric wrote: post #266
ritchiea wrote: post #181

Earlier quoted context omitted.

I believe developers need the courage to do what we feel is necessary and the paranoia to quadruple check our work and assumptions.

Paranoia is often useful, but with a good enviorment, and tools it's rarely need. I find bad assumptions often cause the worst problems. Break things regularly and you end up with fewer assumptions about the code base / production environment which is a vary good thing.

I don't trust tools unless I wrote them myself, in which case I know there will be bugs in them.

Re: How I Fired Myself

#292

Post by capex »

The senior engineers got to own the mistakes of their juniors. That's how teams are built. This clearly didn't happen in this case.

Re: How I Fired Myself

#293

Post by chris_mahan »

Let me make it really simple: Anything that happens in a company is always, always management's fault. The investors hire the management ream to turn a pile of money into a bigger pile of money, and if management fails, it is management's fault, because it can do whatever it needs to do (within the law) to make that happen. That they failed to hire, train, motivate, fire, promote, follow the law, develop the right products, market them well, ensure reliability, ensure business sustainability, ensure reinvestment in research and development, and ultimately satisfy investor, is their fault, and they further demonstrate their failure by not taking responsibility for their own failure and blaming others.

Re: How I Fired Myself

#294

Post by hugoc »

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

I worked for the largest cellphone carrier in my country. I had write permission for the production db (not to all views, though) from my second week there onwards, the first week I used the credentials of the guy training me. The guy training knew the whole thing was wrong, mainly because once he ran a query that froze the db for half day. I was not a developer I was working in the help desk.

Re: How I Fired Myself

#295

Post by sskates »

gte910h wrote: post #179
jcromartie wrote: post #171

Earlier quoted context omitted.

Indeed. If it had been a sporadic hardware failure, they would have been exactly as screwed. The fact that they gave an overworked junior dev direct read/write access to the production database is astounding.

It's not that they gave him r/w at all that's so criminally stupid. It's that the required him to clear a table manually, using generic full access tools, over and over. In reality, this should have been re-factored to the dev db. If it couldn't be, the junior dev should have been given access to the raids table alone for writes. Lastly the developer who didn't back up this table is the MOST to blame. Money was paid…

Yeah I think him manually clearing a table over and over again was the big problem here. The amount of entropy that had to be introduced into the process to turn a routine task into millions of dollars of loss was tiny. He just needed to click in a slightly different spot on the webpage to bring up USERS instead of RAIDS.

Re: How I Fired Myself

#298

Post by hakaaaaak »

cmbaus wrote: post #11

Any manager who doesn't take responsibility for this isn't a manager you'd want to work for. The manager should be fired.

Agreed, but if the manager took responsibility for this he or she probably would be fired. Still, it is the only way to be; otherwise, you're not a real leader.

Re: How I Fired Myself

#299

Post by MBCook »

reinhardt wrote: post #241
happycube wrote: post #225

Earlier quoted context omitted.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

I've seen something like this. It could be a situation where they were transitioning from that backup service to their own server or another service, and the move was never completed due to some hiccup or priority change.

But if no one is watching to ensure the move was finished (or they got distracted), then something many people treat as set-it-and-forget-it could easily get into that state.

Post reply on HN