Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Fired Myself

edu.mkrecny.com

Post reply on HN
Print view View original

381–390 of 424 posts

Re: How I Fired Myself

#381

Post by shaurz »

rodgerd wrote: post #238
downandout wrote: post #223

Earlier quoted context omitted.

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have…

It's the difference between understanding the incident (programmer makes mistake) and the root cause (failing at data integrity 101). Not just no backups, but no audit log of what's happened in the game - I'd expect an MMO to have an append-only event history quite apart from their state information.

At least all transactions for purchased game items should have been logged in a separate database. There's nothing worse for a digital content company than forgetting who bought what.

Re: How I Fired Myself

#382

Post by protomyth »

hef19898 wrote: post #365
protomyth wrote: post #102

Earlier quoted context omitted.

At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.

Interessting, I have been the only guy in my team who had the exact opposite colours. Green was production and red was testing (i didn't do any dev work, so). I guess that came from me being from produciton. But I should have payed more attention when working with other people on my machine, in hindsight... Luckily, nothing bad ever happened!

I liked a green screen (too much time spent with old terminals) and blue was ok to type on but not as nice . I went with the yellow (warning) and red (serious warning) because they are not very comfortable to type on and most people get the "alert" status given Star Trek.

Re: How I Fired Myself

#383

Post by toomuchcoffee »

The CEO leaned across the table, got in my face, and said, "this is a MONUMENTAL fuck up."

It certainly was -- on multiple levels, but ultimately up at the C-level. Blaming a single person (let a lone a junior engineer) for it just perpetuates the inbred culture of clusterfuckitude and cover-ass-ery which no doubt was the true root cause of the fuck-up in the first place.

Re: How I Fired Myself

#384

Post by thebigkick »

reinhardt wrote: post #241
happycube wrote: post #225

Earlier quoted context omitted.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

I don't think it is that far-fetched. I've had experience with a well known company that does millions in revenue per week (web based shopping cart) that just FTP's everything with no DVCS. Designers, developers, managers all have access to the server and db.

Re: How I Fired Myself

#385

Post by mbell »

eloisant wrote: post #347
mbell wrote: post #168

Earlier quoted context omitted.

I'll add another one: * No Foreign Keys Attempting to clear the table should have just thrown a constraint violation error.

Well, depending how you configure your cascade clearing the user table could have cleared all the other tables also :) "on delete cascade"!

True on MySql with InnoDB, wouldn't be true with postgres.

You'd have to use TRUNCATE CASCADE on postgres to avoid the foreign key error.

Re: How I Fired Myself

#386

Post by Aardwolf »

liquidise wrote: post #109
mooreds wrote: post #100

Earlier quoted context omitted.

To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?

The author makes mention of using a UI to connect to their db. If i was in a position over there i can see myself writing a script to clear out the tables i wanted. This reduces errors, but not the risk.

Arrow up, enter Or ctrl+r enter Can be very dangerous if something other than your routine is in the bash history.

Re: How I Fired Myself

#387

Post by Aardwolf »

mehwoot wrote: post #371
readme wrote: post #358

Earlier quoted context omitted.

I wish I could upvote this more. Production database? Seriously, they were one copy away from avoiding this whole outcome. CEO sounds incompetent as hell.

Hindsight is 20/20.

If you have only one copy of data, especially if it is important, the chance of something happening to that copy, either hardware, software, or human error, is always big enough to justify a backup. No hindsight needed for that.

Re: How I Fired Myself

#388

Post by sophacles »

gnarbarian wrote: post #290
cmos wrote: post #107

When I was 18 I took out half my towns power for 30 minutes with a bad scada command. It was my summer job before college and I went from cleaning the warehouse to programming the main SCADA control system in a couple weeks. Alarms went off, people came running in freaking out, trucks started rolling out to survey the damage, hospitals started calling about people on life support and how the backup generators were no…

I'm interested in knowing some more details regarding the architectural setup and organizational structure that would allow something like this to happen.

Honestly, you don't. The IT engineering in power and other SCADA systems is downright scary.

Re: How I Fired Myself

#389

Post by ajross »

PeterisP wrote: post #366
ajross wrote: post #244

Earlier quoted context omitted.

Doing this kind of testing in a gold-plated, heavily-engineered way is hard. But that's not an excuse for not doing it at all. Just walking into your closet and pulling a cable gets you 80-95% of the testing you need, and is free. Setting up a sandbox and "restoring" a backup onto it and then doing some quick queries is likewise easy to do and eliminates huge chunks of the failure space of "bad backups". Really, this…

"walking into your closet and pulling a cable" is not free, if your planned disaster recovery is not a seamless failover, but a process to recover data with some work and limited (nonzero) downtime/cost to business. For example, our recovery plan for a financial mainframe in case of most major disasters was to restore the daily backup to off-site hardware identical to production hw; however, the (expensive) hardware…

To paraphrase: if you go with a awful hack job for your disaster recovery plan, testing is more expensive. And to extend: you won't actually test because it's "too expensive", and your disaster recovery plan won't work.

How is this distinct from "Don't hack your systems to make them work. Absolutely do hack at them to test."? I don't see it.

This just sounds like "my business doesn't have the financial capacity to engineer data recovery processes". Well, OK then. Just don't claim to be doing it.

Re: How I Fired Myself

#390

Post by bfrenchak »

This was not a junior engineer's fault, but the DBA's fault. Any company should be backing up their database regularly, and then testing the restores regularly. Also don't give people access to drop table's etc. This was a very poor setup on the part of the company/DBA not the engineer.
Post reply on HN