Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

371–380 of 424 posts

Re: How I Fired Myself

#371

Post by mehwoot »

readme wrote: post #358
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

I wish I could upvote this more. Production database? Seriously, they were one copy away from avoiding this whole outcome. CEO sounds incompetent as hell.

Hindsight is 20/20.

Re: How I Fired Myself

#372

Post by pc86 »

Domenic_S wrote: post #264
wpietri wrote: post #206

Earlier quoted context omitted.

When moving data from prod to other environments, consider a scrambler. E.g., replace all customer names with names generated from census data. I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody. Since your handle is perlgeek, you're already well equ…

Yep. For x.com I wrote a simple cron job that sterilizes the automated database dump and sends it to the dev server. Roughly, it's like this: -cp the dump to a new working copy -sed out cache and tmp tables -Replace all personal user data with placeholders. This part can be tricky, because you have to find everywhere this lives (are form submissions stored and do they have PII?) -Some more sed to deal with actions/tr…

This sounds like it'd make a good blog post.

Re: How I Fired Myself

#373

Post by geldedus »

it is the fault of those responsible with creating a regular backup procedure and/or a hotswap database server

and developping on the production database speaks volumes about the incompetency level of that company and of the "developer" in particular, afterall

Re: How I Fired Myself

#374

Post by hga »

pja wrote: post #360
geoka9 wrote: post #312

Earlier quoted context omitted.

Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth.... I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.

Maybe the clerk was carefully sealing them back up again? In retrospect they probably should have had tamper-proof seals given the value attached to the wafers being delivered unopened, but then most problems are easily avoidable in retrospect.

Exactly; this happened early enough in the history of ICs that I'm sure they weren't taking such precautions, just like tamper revealing seals for over the counter drugs didn't become big until after the Chicago Tylenol murders.

There's a good chance a tamper revealing seal would have stopped the clerk from opening the containers, and of course it they'd been broken before reaching the people at the fab lines who were supposed to open them that would have clued Intel into the problem before any went into production and would have allowed them to quickly trace the problem back upstream.

Re: How I Fired Myself

#375

Post by hga »

sybhn wrote: post #234
bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Absolutely. Your immediate technical management sucked, and you were made the scape goat for your management's failure. Welcome to the real world. Don't get me wrong, you should feel bad, very bad, bad enough so that never again you do that. But you shouldn't feel guilty nor rethink your career.

A little bit of feeling guilty is in order; the author "didn't know that he didn't know", and I'm sure this motivated him to learn a lot more about proper engineering processes ... something I'll note aren't particularly a focus in CS degrees. Especially since I haven't come across anyone who's really dedicated to them who hadn't first gotten burned in one way or another.

There's a big difference between being told "Do X, don't do Y" and that sinking feeling you get when you realize a big problem exists, regardless of the eventual outcome.

Re: How I Fired Myself

#376

Post by hga »

deservingend wrote: post #329
hga wrote: post #162

Earlier quoted context omitted.

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…

Seems like sabotage.

"Never attribute to malice that which is adequately explained by stupidity." (http://en.wikipedia.org/wiki/Hanlons_razor)

Or in this case energy, ignorance, and not learning enough of the big picture, or wondering why these wafers were sealed in air tight containers in the first place.

It's the well meaning mistakes that tend to be the most dangerous since most people are of good will, or at least don't want to kill the company and lose their jobs.

Re: How I Fired Myself

#378

Post by stevekemp »

mappu wrote: post #319
stevekemp wrote: post #259

Earlier quoted context omitted.

Turning off the wrong-server is a thing that bit me before I installed molly-guard. These days that, and similar, is a tweak I apply to all hosts I control. (molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)

Steve Kemp? Slightly unrelated, but i just want to say i'm (still) a big fan of your old 2004 program window.exe. Very handy for unhiding the odd broken program. Have a wonderful day! (and i'll definitely look at installing molly-guard on my production debian servers)

Yes, this is dog! Small world. :)

Re: How I Fired Myself

#379

Post by ramchip »

ghjm wrote: post #154
paganel wrote: post #140

Earlier quoted context omitted.

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage". Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger tea…

This is correct and is the way it should be. So how come the programmers are always politically gunning for the keys to the production server cabinet, where you do have to be afraid to break things?

Are they? Where I work (a bank) we were more than happy to move to read-only access to prod servers, and pass by a support team when we need to deploy things.

Re: How I Fired Myself

#380

Post by shaurz »

It's an organisational failure if a junior employee can bring down the company in a few clicks. No backups, testing on the production database, this is no way to run a company. Feel sorry for the guy who made a simple mistake.
Post reply on HN