Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

281–290 of 424 posts

Re: How I Fired Myself

#281

Post by amputect »

chris_wot wrote: post #235
Cthulhu_ wrote: post #149

Earlier quoted context omitted.

And yet, the net result is the same; right-click, clear table, or ./clearTable.sh. Both are human actions, and both are fallible. What if some prankster edited clearTable.sh to do the users table instead of the raids table? What if he did it himself to test something?

Forget a prankster! What happens if the full RAID array fails that holds the database? No backups: dead company.

Heck when you put it that way, this guy actually did them a FAVOR. He ONLY wiped out the User table. The company was able to learn the value of backups, and they had enough data left to be able to partially recover it from the remaining tables, which is much better than the worst case scenario.

Re: How I Fired Myself

#282

Post by hkmurakami »

I'm. a little worried for OP because he obviously took the time to keep the characters in this article anonymous, but we now know who this CEO with ridiculous behavior must have been, since we know the name of OP's former company from his profile. Not sure what said former CEO of Noe acquired company can do, but this is the kind of thing I fear happeneing toe when/if I write something negative about a past employer, being a blogger myself.

Re: How I Fired Myself

#283

Post by hkmurakami »

I'm. a little worried for OP because he obviously took the time to keep the characters in this article anonymous, but we now know who this CEO with ridiculous behavior must have been, since we know the name of OP's former company from his profile. Not sure what said former CEO of Noe acquired company can do, but this is the kind of thing I fear happeneing toe when/if I write something negative about a past employer, being a blogger myself.

Re: How I Fired Myself

#285

Post by damm »

gnur wrote: post #251
bguthrie wrote: post #205

Earlier quoted context omitted.

The problem isn't who has the keys , it's how they're used . I don't care as much if a junior developer has the prod password; I care more about building an engineering and ops team that understands that dicking around with the prod database isn't okay . Sysops and DBAs are fallible too--I've seen a lot of old school shops that relied heavily on manual migration and configuration. Automate, test, isolate and expect f…

That's why you don't give out the keys to people that know how to act responsible.

Don't you mean that's why you don't give keys to people that do not know how to act responsibly?

Giving keys to irresponsible people seems irresponsible ;)

Re: How I Fired Myself

#286

Post by smoyer »

grey-area wrote: post #270
hga wrote: post #220

Earlier quoted context omitted.

Bingo. As I like to emphasize, people don't care about backups---this company certainly didn't---they care about restores . And almost no one is willing to put up the money to do compete testing of restore paths, let along statistically making sure they continue to work.

My favourite way to test restores is to do them frequently to the dev server from the production backups - this keeps the dev data set up to date, and works as a handy test of the restore mechanism. Of course if you have huge amounts of data or files on production this becomes more difficult, but not impossible, to manage.

This works well, though you may need an "anonymizer" (and maybe some extra compliance testing) if your systems have PCI or HIPPA data on them. We have federal restrictions against storing certain types of data on servers outside the US. Cloud computing sounds great but neither Amazon or Google will guarantee the data stays within the country's borders.

Re: How I Fired Myself

#287

Post by dntrkv »

angersock wrote: post #261
reinhardt wrote: post #241

Earlier quoted context omitted.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

Having just helped a friend who was miffed at the idea of spending money on a new UPS for >$5k worth of networking equipment: do not be surprised. Penny-wise pound-foolish.

Spending $5k on a UPS is very different from not having backups of your production database which runs your multimillion dollar business. This story just doesn't add up.

Re: How I Fired Myself

#288

Post by coopdog »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

That's the Agile way, people not processes.

When the people screw up, there's no processes to hold them back, they can really, really screw it up.

Once you hit millions in revenue it's probably wise to put a couple of fall back processes in place, reliability becomes as important as agility.

Re: How I Fired Myself

#289

Post by potatolicious »

j_baker wrote: post #214
potatolicious wrote: post #124

Earlier quoted context omitted.

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used…

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

I wish I could say I have some sort of prescience about bad working environments. In reality I'm just good at getting out while the gettin' is still good.

Re: How I Fired Myself

#290

Post by gnarbarian »

cmos wrote: post #107

When I was 18 I took out half my towns power for 30 minutes with a bad scada command. It was my summer job before college and I went from cleaning the warehouse to programming the main SCADA control system in a couple weeks. Alarms went off, people came running in freaking out, trucks started rolling out to survey the damage, hospitals started calling about people on life support and how the backup generators were no…

I'm interested in knowing some more details regarding the architectural setup and organizational structure that would allow something like this to happen.
Post reply on HN