Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

311–320 of 424 posts

Re: How I Fired Myself

#311

Post by lukeschlather »

smoyer wrote: post #286
grey-area wrote: post #270

Earlier quoted context omitted.

My favourite way to test restores is to do them frequently to the dev server from the production backups - this keeps the dev data set up to date, and works as a handy test of the restore mechanism. Of course if you have huge amounts of data or files on production this becomes more difficult, but not impossible, to manage.

This works well, though you may need an "anonymizer" (and maybe some extra compliance testing) if your systems have PCI or HIPPA data on them. We have federal restrictions against storing certain types of data on servers outside the US. Cloud computing sounds great but neither Amazon or Google will guarantee the data stays within the country's borders.

Have you looked at https://aws.amazon.com/govcloud-us/ ?

Re: How I Fired Myself

#312

Post by geoka9 »

hga wrote: post #162
columbo wrote: post #21

News flash, If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?" They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are. As an aside: Way back…

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?" In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory: One day Intel's yields suddenly went to…

Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....

I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.

Re: How I Fired Myself

#313

Post by cybernomad99 »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

To the credit of the management, they did not fire him. He resigned. But the coworkers felt he was responsible personally. That makes a uneasy work environment.

Re: How I Fired Myself

#314

Post by jpalacios »

clavalle wrote: post #252
jpalacios wrote: post #135

Earlier quoted context omitted.

Yah this story made me cringe. What exactly do you mean by: "Using a relational database as a flat data store? Super bad." Are you referring to the users table? I am not too accustomed to using flat files, so I am curious.

Users is a bit of a core table in most applications. If they were using the relational database as it should be used there would be references to the user table elsewhere in the database. If you tried to delete the table, it would fail stating that a deletion would violate the constraints assuming you didn't have deletions cascade automatically (which would be equally bad). On the other hand (and it probably happened…

Ah I see.. I misunderstood the first time around. I thought you meant to store the user table in a flat file. Thank you for the explanation. That reminds me, I need to convert to Innodb one of these days.

Re: How I Fired Myself

#315

Post by mcherm »

sixtypoundhound wrote: post #243
yaj wrote: post #183

Earlier quoted context omitted.

This is the right thing to encourage but I just would like to add always have a backup. "Don't be afraid to break things as long as you have a backup". It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.

I would go one step farther.... Every production deployment should involve blowing away the prior instance, rebuilding from scratch, and restarting the service; you are effectively doing a near-full "restore" for every deployment, which forces you to have everything fully backed up and accessible... Any failure to maintain good business continuity practices will manifest early for a product / employee / team, which a…

Spoken like a man who has maintained applications but never databases.

In the world where data needs to be maintained, this is not necessarily an option. In the bank where I work, we deploy new code without taking any outage (provide a new set of stored procedures in the database, then deploy a second set of middleware, then a new set of front-end servers, test it, then begin starting new user sessions on the new system; when all old user sessions have completed the old version can be turned off). Taking down the database AT ALL would require user outages. Restoring the database from backup is VERY HARD and would take hours (LOTS of hours).

That being said, we do NOT test our disaster-recovery and restore procedures well enough.

Re: How I Fired Myself

#316

Post by gtirloni »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

When I was a junior analyst, I once deleted the main table that contained all 70k+ users, passwords, etc. The problem was fixed in 15min after the DBA was engaged to copy all data back from the QA environment that was synchronized every X minutes. Or we could have restored a back from a few hours ago.

The whole company fucked this one up pretty badly. NO excuses.

Re: How I Fired Myself

#317

Post by rurounijones »

protomyth wrote: post #102
hollerith wrote: post #82

Earlier quoted context omitted.

To avoid mistakes like that is why I put the hostname and only the hostname plus one character in my shell prompt. (The other character is a # or $ depending on whether the user is root or not.)

At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.

I have done this on a few servers but found that it always screws up formatting of the lines in bash when they are long and you are hitting up and going back through the history.

Did you change the $PS1 variable? Can you share your config?

Re: How I Fired Myself

#318

Post by jliptzin »

bguthrie wrote: post #86

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

Unbelievable. Even at my 3 person startup, back in 2010, with thousands in revenue, not millions, we had development environments with test databases and automated daily database snapshotting. Sure I've accidentally truncated a few tables in my time, but luckily I wasn't dumb enough to be developing on a production server.

Re: How I Fired Myself

#319

Post by mappu »

stevekemp wrote: post #259
islon wrote: post #18

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a…

Turning off the wrong-server is a thing that bit me before I installed molly-guard. These days that, and similar, is a tweak I apply to all hosts I control. (molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)

Steve Kemp? Slightly unrelated, but i just want to say i'm (still) a big fan of your old 2004 program window.exe. Very handy for unhiding the odd broken program.

Have a wonderful day! (and i'll definitely look at installing molly-guard on my production debian servers)

Re: How I Fired Myself

#320

Post by krob »

damm wrote: post #285
gnur wrote: post #251

Earlier quoted context omitted.

That's why you don't give out the keys to people that know how to act responsible.

Don't you mean that's why you don't give keys to people that do not know how to act responsibly? Giving keys to irresponsible people seems irresponsible ;)

That's why you avoid IT like the plague and use Heroku, that why when someone Fubar's you can blame it on amazon web services :) j/k
Post reply on HN