261–270 of 424 posts
Earlier quoted context omitted.
And worse, they had a backup service, and then dropped it to save money.
This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".
Penny-wise pound-foolish.
If the company saw that the data contained in this live database was so critical you would have thought that would not have given the keys to everyone and that if they did, they would at least make sure that they can recover from this, and fast.
1) Senior developers / CTO letting anybody mess with the prod DB should be grounds for their firing. It's so incompetent, it's insane.
2) No backups. How is this even possible. You even had paying customers.
Earlier quoted context omitted.
Reading those stories makes me realize how well thought-out the process at my work is: We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the stag…
When moving data from prod to other environments, consider a scrambler. E.g., replace all customer names with names generated from census data. I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody. Since your handle is perlgeek, you're already well equ…
-cp the dump to a new working copy
-sed out cache and tmp tables
-Replace all personal user data with placeholders. This part can be tricky, because you have to find everywhere this lives (are form submissions stored and do they have PII?)
-Some more sed to deal with actions/triggers that are linked to production's db user specifically.
-Finally, scp the sanitized dump to the dev server, where it awaits a Jenkins job to import the new dump.
The cron job happens on the production DB server itself overnight (keeping the PII exposure at the same level it is already), so we don't even have to think about it. We've got a working, sanitized database dump ready and waiting every morning, and a fresh prod-like environment built for us when we log on. It's a beautiful thing.
Earlier quoted context omitted.
I'd like to see a source for Romney outspending the Obama team "at least" 10x, because while I can speak from experience that ORCA was a gigantic piece of shit, it's not like the Obama people were struggling to pay their bills.
I don't know what metric the parent comment is referring to, but in terms of technology stack, I can fully believe that the Romney team spent more than Obama's team. Here's a post by one of the creators of the fundraising platform: http://kylerush.net/blog/meet-the-obama-campaigns-250-millio... The short of it is: they used static HTML generated by Jekyll and stored on S3.
> ... what the Obama team did for their Narwhal infrastructure - they staged outages and random failures to prepare for their big day, meanwhile Romney's team who outspent the Obama team at least an order of magnitude, had their system fail on e-day.
Earlier quoted context omitted.
Heh. Another way I would say that is that "discretion is the better part of valor" And, to rip-off Hitchhiker's Guide to the Galaxy, "cowardice is the better part of discretion"...in that if fear makes you judiciously check your backups and write tests, then that's not a bad thing at all.
I believe developers need the courage to do what we feel is necessary and the paranoia to quadruple check our work and assumptions.
Tens of thousands of paying customers and no backups ? No staging environment (from which ad-hoc backups could have been restored)!?!? No regular testing of backups to ensure they work? No local backups on dev machines?!? Using a GUI tool for db management on the live db?!?!? No migrations!?!?! Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?! What an astonishing failure of process. The…
I still use the mysql CLI and have for 10 years plus-or-minus, but I actually use Sequel Pro a lot. If I'm perusing tables with millions of rows, or I want to quickly see a schema, or whatever, it's been a net gain in productivity.
In addition, to heal your pain its best to hear that you're not the only one who has ever done this. Trust me, all engineers I know have a story like this. (Please share yours HN - Here I even started a thread for it: http://news.ycombinator.com/item?id=5295262)
Here is mine: When I worked for a financial institution my manager gave me a production level username and password to help me get through the mounds of red tape which usually prevented any real work from getting done. We were idealists at the time. Well I ended up typed that password wrong, more than 3 times...shit, I locked the account. Apparently half of production's apps were using this same account to access various parts of the network. Essentially, I brought down half our infrastructure in one afternoon.
Lesson learned: Don't use the same account for half your production apps. Not really my fault :).
Earlier quoted context omitted.
Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected. What's more questionable is: * Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter. * Junior developers don't need an access to production machine, only sysops and maybe the technical PM. * No bac…
I'll add another one: * No Foreign Keys Attempting to clear the table should have just thrown a constraint violation error.
Earlier quoted context omitted.
This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a net…
Bingo. As I like to emphasize, people don't care about backups---this company certainly didn't---they care about restores . And almost no one is willing to put up the money to do compete testing of restore paths, let along statistically making sure they continue to work.