Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

151–160 of 424 posts

Re: How I Fired Myself

#151

Post by hga »

bambax wrote: post #84

Once, a long time ago, I spent the best part of a night writing a report for college, on an Amstrad PPC640 ( http://en.wikipedia.org/wiki/PPC_512 ). Once I was finished, I saved the document -- "Save" took around two minutes (which is why I rarely saved). I had an external monitor that was sitting next to the PC; while the saving operation was under way, I decided I should move the monitor. The power switch was on to…

Yeah; I was lucky that my first experience where I could lose data like that (before it was on punched cards) was a nice UNIX(TM) V6 system on a PDP-11/70 that had user accessible DECTAPEs. Because I found the concept interesting, I bought one tape, played around with it including backing up all my files ... and then I learned the -rf flags to rm ^_^.

That was back in the summer of 1978; today I have an LTO-4 tape drive driven by Bacula and backup the most critical stuff to rsync.net, the latter of which saved my email archive when the Joplin tornado roared next to my apartment complex and mostly took out a system I had next to my balcony sliding glass doors and the disks in another room with my BackupPC backups.

As long as we're talking about screwups, my ... favorite was typing kill % 1, not kill %1, as root, on the main system the EECS department was transitioning to (that kills the initializer "init", from which all child processes are forked). Fortunately it wasn't under really serious heavy use yet, but it was embarrassing.

Re: How I Fired Myself

#152

Post by Morendil »

mkrecny wrote: post #54
lallouz wrote: post #31

I would love to see some reflection on this story from OP. What do you think you learned from this experience? Do you think your response was appropriate? What would you have done differently? Are you forever afraid of Prod env? Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.

OP here. I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.

See this: http://www.slideshare.net/danmil30/how-to-run-a-5-whys-with-...

Guilt is a moral concept; when it comes to a run-of-the-mill operations mistake like yours, it does not belong in analysis of its consequences to the business. You are not a robot. You have made and will make mistakes this bad and worse.

Only consequentialist thinking should be the order of the day here; "what do we know that can prevent a similar mistake from hurting our bottom line". In this case backups are the standard, reasonable, well-known practice. Nothing will be improved by a firing or a resignation, by blaming or by shaming.

Insofar as the real root cause of the problem was not addressed, it's a reasonable prediction that any such company eventually joins the deadpool due to similar oversights.

Re: How I Fired Myself

#153

Post by x3ord »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Exactly. I'm ashamed that my first reaction reading this was to blame OP. But in the 2 min it took to read the post I had come full circle to wondering what kind of terribly run company would allow this to happen--I guess the type that hires philosophy majors straight out of college without vetting their engineering skills.

Re: How I Fired Myself

#154

Post by ghjm »

paganel wrote: post #140
ritchiea wrote: post #106

Earlier quoted context omitted.

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental. I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev ne…

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

This is correct and is the way it should be. So how come the programmers are always politically gunning for the keys to the production server cabinet, where you do have to be afraid to break things?

Re: How I Fired Myself

#155

Post by rocky1138 »

mootothemax wrote: post #71
danielh wrote: post #63

Earlier quoted context omitted.

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data. Excellent point! Any tips on how to avoid that, other than not taking the data home / copying to personal Dropbox-type things?

I've done this before and I just email it to myself using the company email account. This way nothing leaves the workplace. Also, no financial transaction data was in the db as it was a simple wordpress blog.

If it stored credit card data or other important stuff I'd take a look at what PCI compliance says you have to do for your backups and follow that.

Re: How I Fired Myself

#156

Post by ricardobeat »

stephengillie wrote: post #144
lsc wrote: post #90

Earlier quoted context omitted.

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said. (and if the CTO, or whoever is in…

I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it. Does Amazon Glacier offer this?

I think since it uses generated IDs for each archive, it's impossible to overwrite anything.

Re: How I Fired Myself

#157

Post by rocky1138 »

mootothemax wrote: post #57

If you ever notice that your employer or client isn't backing up important data, take a tip from me: do a backup, today, in your free time, and if possible, and again in your free time, create the most basic regular backup system you can. When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.

This.

Even if it's old, when you're facing no data or old data, old data looks like heaven.

Re: How I Fired Myself

#158

Post by anovikov »

LOL a gaming startup i worked for in 2010 had the same fuckup! But nobody was fired or quit, there was just a total anger around the place for a few days, and almost all data was eventually recovered. The startup still flopped in about one year after that with ever falling user retention rates - the marketplace was more and more flooded with those more and more similar games.

Re: How I Fired Myself

#159

Post by jakejake »

grey-area wrote: post #108

Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…

My hypothesis is that it's a game company and all of the focus was on the game code. The lowly job of maintaining the state server was punted off to the "junior dev" just out of school. Nobody was paying attention. It was something that just ran.

They paid the price of ignoring what was actually the most critical part of their business.

Re: How I Fired Myself

#160

Post by heathlilley »

asveikau wrote: post #143
mooreds wrote: post #100

Earlier quoted context omitted.

To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?

The junior dev was 22 at the time and fresh out of school. I'm assuming the senior devs were 23 and had a year on the job. The principal devs were a ripe 24, otherwise known as old men. :-)

He and his coworkers might not have been "Senior" before and they still might not be fully senior, but they are MUCH closer now.

>Ding! >Gratz!

Post reply on HN