Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

231–240 of 424 posts

Re: How I Fired Myself

#231

Post by hef19898 »

xentronium wrote: post #16

This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible. Anecdote: I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important lat…

Remebers me of adifferent story from back the day when I worked around a pretty huge SAP system (as some kind of super user whatever). In one, seemingly trivial update (trivial compared to the complete system upgrade from one version to the higher one that worked without problems) cleansed the database including all purchase orders from the last two days company wide. Ah, and the back-up becamo "unusable", too.

But as far as I know, nobody was fired for this. Because, yes, things like this just can happen. An eventually it got fixed anyway.

Re: How I Fired Myself

#232

Post by Yare »

If it helps explain things, the only experience the CEO had before this social game shop was running a literal one-man yogurt shop.

This happened a week before I started as a Senior Software Engineer. I remember getting pulled into a meeting where several managers who knew nothing about technology were desperately trying to place blame, figure out how to avoid this in the future, and so on.

"There should have been automated backups. That's really the only thing inexcusable here.", I said.

The "producer" (no experience, is now a director of operations, I think?) running the meeting said that was all well and good, but what else could we do to ensure that nobody makes this mistake again? "People are going to make mistakes", I said, "what you need to focus on is how to prevent it from sinking the company. All you need for that is backups. It's not the engineer's fault.". I was largely ignored (which eventually proved to be a pattern) and so went on about my business.

And business was dumb. I had to fix an awful lot of technical things in my time there.

When I started, only half of the client code was in version control. And it wasn't even the most recent shipped version. Where was the most recent version? On a Mac Mini that floated around the office somewhere. People did their AS3 programming in notepad or directly on the timeline. There were no automated builds, and builds were pushed from peoples' local machines -often contaminated by other stuff they were working on. Art content live on our CDN may have had source (PSD/FLA) distributed among a dozen artist machines, or else the source for it was completely lost.

That was just the technical side. The business/management side was and is actually more hilarious. I have enough stories from that place to fill a hundred posts, but you can probably get a pretty good idea by imagining a yogurt-salesman-cum-CEO, his disbarred ebay art fraudster partner, and other friends directing the efforts of senior software engineers, artists, and other game developers. It was a god damn sitcom every day. Not to mention all of the labor law violations. Post-acquisition is a whole 'nother anthology of tales of hilarious incompetence. I should write a book.

I recall having lunch with the author when he asked me "What should I do?". I told him that he should leave. In hindsight, it might have been the best advice I ever gave.

Re: How I Fired Myself

#233

Post by groby_b »

ajross wrote: post #211
indrax wrote: post #184

Earlier quoted context omitted.

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…

Doing that kind of testing is hard. It costs time and effort. If you want to see it done on a truly awe-inspiring scale (whole data centers being taken down by zombies ;) : http://queue.acm.org/detail.cfm?id=2371516

Re: How I Fired Myself

#234

Post by sybhn »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Absolutely. Your immediate technical management sucked, and you were made the scape goat for your management's failure. Welcome to the real world. Don't get me wrong, you should feel bad, very bad, bad enough so that never again you do that. But you shouldn't feel guilty nor rethink your career.

Re: How I Fired Myself

#235

Post by chris_wot »

Cthulhu_ wrote: post #149
liquidise wrote: post #109

Earlier quoted context omitted.

The author makes mention of using a UI to connect to their db. If i was in a position over there i can see myself writing a script to clear out the tables i wanted. This reduces errors, but not the risk.

And yet, the net result is the same; right-click, clear table, or ./clearTable.sh. Both are human actions, and both are fallible. What if some prankster edited clearTable.sh to do the users table instead of the raids table? What if he did it himself to test something?

Forget a prankster! What happens if the full RAID array fails that holds the database? No backups: dead company.

Re: How I Fired Myself

#236

Post by ismarc »

jiggy2011 wrote: post #199
ismarc wrote: post #104

Earlier quoted context omitted.

Part of our culture/personality/team fit questions is like this. We also have one that's something like "tell me about a time you failed a commitment (deadline, etc) and how you handled it". Numerous people with over a decade claim perfect records, frequently blaming all those around them as having failed. It's been a real easy way to eliminate candidates, particularly because almost every team member we have gave an…

Interesting, though perhaps they misinterpreted the question. Some companies seem to have a culture where you never admit failure and perhaps they assumed that was the answer you were looking for?

All of the failed responses weren't just not admitting failure, but directly blaming others. One example was a team lead in charge of a project where they said the developers missed their deadlines and would "whine" about not having enough details to do their job. Someone who didn't have an example of failing to deliver, but had examples of near misses they were able to save would even be good examples. Playing the blame game, though, just doesn't fit.

Re: How I Fired Myself

#237

Post by Qantourisc »

fusiongyro wrote: post #37
eloisant wrote: post #25

Earlier quoted context omitted.

Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected. What's more questionable is: * Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter. * Junior developers don't need an access to production machine, only sysops and maybe the technical PM. * No bac…

It's a nice theory, but not everybody has that much staff. You can get by with proper backups and running against local copies of production.

Constrains also act like bug protections. If you screw up your database tells you right away, and not a beta tester much later.

Re: How I Fired Myself

#238

Post by rodgerd »

downandout wrote: post #223
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have…

It's the difference between understanding the incident (programmer makes mistake) and the root cause (failing at data integrity 101). Not just no backups, but no audit log of what's happened in the game - I'd expect an MMO to have an append-only event history quite apart from their state information.

Re: How I Fired Myself

#239

Post by danso »

pc86 wrote: post #222
r00fus wrote: post #218

Earlier quoted context omitted.

Apple did this before Amazon or Netflix in this regard [1], but the point needs to be made that a system needs to be tested and not just in a controlled aseptic way, because the real world isn't. Another story supporting Chaos Monkey is what the Obama team did for their Narwhal infrastructure - they staged outages and random failures to prepare for their big day, meanwhile Romney's team who outspent the Obama team at…

I'd like to see a source for Romney outspending the Obama team "at least" 10x, because while I can speak from experience that ORCA was a gigantic piece of shit, it's not like the Obama people were struggling to pay their bills.

I don't know what metric the parent comment is referring to, but in terms of technology stack, I can fully believe that the Romney team spent more than Obama's team. Here's a post by one of the creators of the fundraising platform:

http://kylerush.net/blog/meet-the-obama-campaigns-250-millio...

The short of it is: they used static HTML generated by Jekyll and stored on S3.

Re: How I Fired Myself

#240

Post by HeyLaughingBoy »

wpietri wrote: post #202
gefh wrote: post #182

Earlier quoted context omitted.

I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...

We are all born naked, bloody, and screaming; the only thing we know is how to work a nipple. Everything else has to be learned. One of Toyota's mantras is "If the student has failed to learn, the teacher has failed to teach." Their point is that managers are responsible for solving issues that come from employee ignorance, not line workers.

Exactly. In my organization I know of perfectly good hardware that is either being tossed or used for non-critical applications because someone didn't follow the Incoming Inspection process correctly. It doesn't matter that they could simply be inspected now and found to be perfect, the process wasn't followed, so the product is "junk."
Post reply on HN