How I Fired Myself • Hacker News Board

Re: How I Fired Myself

#351

Post by Nick_C » Thu, Feb 28, 2013, 9:58 AM UTC

hollerith wrote: post #82

islon wrote: post #18

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a…

To avoid mistakes like that is why I put the hostname and only the hostname plus one character in my shell prompt. (The other character is a # or $ depending on whether the user is root or not.)

I also change the prompt colour according to server. I can tell at a glance which machine I'm on (plus the hostname of course).

It means having a slightly difference .bashrc for each machine, but it's trivial.

Re: How I Fired Myself

#352

Post by dsuth » Thu, Feb 28, 2013, 9:59 AM UTC

amputect wrote: post #281

chris_wot wrote: post #235

Earlier quoted context omitted.

Forget a prankster! What happens if the full RAID array fails that holds the database? No backups: dead company.

Heck when you put it that way, this guy actually did them a FAVOR. He ONLY wiped out the User table. The company was able to learn the value of backups, and they had enough data left to be able to partially recover it from the remaining tables, which is much better than the worst case scenario.

Do you think the company learned the value of backups, or do you think they learned to blame junior devs for fuckups? Sounds like they learned nothing, because no attempt was made to determine the root cause.

Re: How I Fired Myself

#353

Post by ThePhysicist » Thu, Feb 28, 2013, 10:00 AM UTC

The only people that should have gotten fired for this are:

* The person responsible for the database backup (no backup plan for your production DB!? wtf)

* The person having designed the SQL admin tool (not putting an irreversible DELETE operation behind a confirmation dialogue!? wtf)

* The person giving full write access to the company's production database to a junior developer (data security!? wtf)

Sure, the employee made a mistake, but most of the failure here is due to the bad management and bad organizational design.

Re: How I Fired Myself

#354

Post by deciob » Thu, Feb 28, 2013, 10:16 AM UTC

This is the most incredible story I have read in a long time. To have such a business relying all on one database and no backups... unbelievable!

Re: How I Fired Myself

#355

Post by alyrik » Thu, Feb 28, 2013, 10:33 AM UTC

Oh dear... I once logged into the postgresql database of a very busy hosted service in order to manually reset a user's password. So I started to write the query:

UPDATE principals SET password='

Then I went and did all the stuff required to work out the correctly hashed and salted password format, then finally triumphantly pasted it in, followed by '; and newline.

FORGOT THE WHERE CLAUSE.

(Luckily, we had nightly backups as pg_dump files so I could just find the section full of "INSERT INTO principals..." lines and paste in a rename of the old table, the CREATE TABLE from the dump, and all the INSERT INTOs, and it was back in under a minute - short enough that anybody who got a login failure tried again and then it worked, as we didn't get any phonecalls). It was a most upsetting experience for me, however...

Re: How I Fired Myself

#356

Post by Tichy » Thu, Feb 28, 2013, 10:47 AM UTC

Just wondering, when consulting I usually take care that there are appropriate clauses in the contract to make me not liable. But what is the rule for employees, are they automatically insured?

In Germany there is the concept of "Fahrlässig" (negligence) and "severe negligence". Per law you are already liable if you are just negligent, but it is possible to lower it to severe negligence in the contract. That is my understanding anyway (not a lawyer). Usually I also try to kind of weasel out of it by saying the client is responsible for appropriate testing and stuff like that... Overall it is a huge problem, though, especially if the client has a law department. Getting insurance is quite expensive because it's easy to create millions of dollars in damages in IT.

Before court "standard best practices" can become an issue, too. This worries me because I don't agree with all the latest fads in software development. It seems possible that in the future x% test coverage could be required by law, for example. Or even today a client could argue that I didn't adhere to standard best practices if I don't have at least 80% test coverage (or whatever, not sure what a reasonable number would be).

Re: How I Fired Myself

#357

Post by zaidf » Thu, Feb 28, 2013, 10:48 AM UTC

It seems insane that you still worked 3 days in a row after a gigantic mistake that can be attributed in good part to being overworked.

Once the damage was done, I would have sent you home instead of overworking you further.

Re: How I Fired Myself

#358

Post by readme » Thu, Feb 28, 2013, 10:51 AM UTC

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

I wish I could upvote this more. Production database? Seriously, they were one copy away from avoiding this whole outcome.

CEO sounds incompetent as hell.

Re: How I Fired Myself

#359

Post by mac1175 » Thu, Feb 28, 2013, 11:08 AM UTC

Wow. This reminds me of a time in which I used to work for a consulting agency. It was back in 2003 and I was working on a some database development for one of the company's biggest clients. One day, I noticed the msdb database had a strange icon telling me it was corrupted. I went onto MSDN and followed some instructions to fix it and, BAM, the database I was working for months on was gone (I was running SQL Server 2000 locally where this all happened and I was very junior as a SQL developer). I was silently freaking out knowing this could cost me my job. I got up from my desk and took a walk. On that walk, I contemplated my resignation. When I got back from my walk, a thought occurred to me that maybe the database file is still there (I had zero clue at the time that msdb's main purpose was just cataloguing the existing databases among other things). I did a file search in the MSSQL folders and found a file named with my database's name. So, that day I learned to attach a database, what msdb's role is, and to make sure to take precautions before making a fix! However, OP's post shows that this company had no processes in place control levels of access or disaster recovery. Show the company's faults more than OP's.

Re: How I Fired Myself

#360

Post by pja » Thu, Feb 28, 2013, 11:45 AM UTC

geoka9 wrote: post #312

hga wrote: post #162

Earlier quoted context omitted.

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…

Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth.... I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.

Maybe the clerk was carefully sealing them back up again? In retrospect they probably should have had tamper-proof seals given the value attached to the wafers being delivered unopened, but then most problems are easily avoidable in retrospect.