41–50 of 119 posts
It kind of makes me wish NILFS2 would become production-ready faster. Give MySQL its own partition, and just roll back to a previous checkpoint if you wipe everything. Not a substitute for backups, but a pretty speedy way to recover for a minor snafu like this.
But if you're logging stuff, NILFS rocks: http://lists.luaforge.net/pipermail/kepler-project/2009-June...
Earlier quoted context omitted.
Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second e…
Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at some point bring down the whole business, perhaps you should ponder it some more.
Actually, screw a book... Does anyone else want to start some wiki pages their experiences with screw-ups, the causes and the solutions? Does this exist in a comprehensive way and I just haven't found it?
If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…
(It's not possible to do this in every case, especially if you make heavy use of stored procedures and triggers, but I don't. If I need client-independent behavior or integrity checks on top of the database, I just use a small RPC server. This makes testing and scaling easier, since there are just simple components that speak over the network. Much easier than predicting everything that could possibly happen to the database.)
I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.
I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…
Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).
Earlier quoted context omitted.
> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second e…
I have a strict rule for myself: never use any curse words in any comments, variable names, dummy accounts etc.
Earlier quoted context omitted.
In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for…
I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care. It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a…