Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.
Whoa.
Devops Horror Stories
21–30 of 102 posts
Re: Devops Horror Stories
#22"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?
[deleted]
Re: Devops Horror Stories
#23Are there any open source load balancing solutions like what Amazon ELB does? Say, install the load balancer on to one or two Amazon VPS, proxy traffic to third party VPS/dedicated servers, Linode, OVH, etc. Wonder how feasible this approach is?
Among different datacenters this is done most often in DNS in order to avoid the increased latency and singe-point-of-failure.
Re: Devops Horror Stories
#24My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.
I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code.
Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.
Re: Devops Horror Stories
#25"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?
The problem was with their temporary storage on a local filesystem, not S3. I'm sure there's some kind of limit to what S3 will allow you to store based on how they distribute data to servers, but 64k isn't it.
Re: Devops Horror Stories
#26My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.
I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.
Yeah, with "real" CI which some people seem to hate, checking in bad code becomes the problem, which seems way better than just waiting to deploy.
Re: Devops Horror Stories
#27Are there any open source load balancing solutions like what Amazon ELB does? Say, install the load balancer on to one or two Amazon VPS, proxy traffic to third party VPS/dedicated servers, Linode, OVH, etc. Wonder how feasible this approach is?
It's not about the load balancing software itself, say HAProxy or Nginx, but with ELB AWS autoscales and handles failover between availability zones. You could certainly handle spinning up your own LB instances, managing DNS/Elastic IPs to handle failovers, etc. It would be far more expensive in setup time, management time and EC2 bill, than ELB which is practically free, starting at less than $20/month.
Re: Devops Horror Stories
#28My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.
I built a system where our developers can do instant deployments of any of our software packages (and instant point-in-time rollbacks), and then do zero-downtime restarts of services. Now we deploy dozens of times a day and I never get called on a Friday night because someone did something stupid. Edit: I do get called when I did something stupid and it broke the deployment system. But that's gotten much rarer lately…
Would love to hear more about this system, sounds really cool!
Re: Devops Horror Stories
#29The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.
Other than the bridge problem with OVH a month back, OVH has been fairly stable network-wise for me for the past year or so. And I rely on the network a ton since I run distributed crawlers.
Re: Devops Horror Stories
#30Tape Archive System: write a tape, restore it again, and do MD5sum against the original data. Then we know it can be restored correctly, and the original data is deleted.
Should be bullet proof?
Alas, the 'write to tape' scripts I'd inherited didn't warn if they couldn't load a tape into the drive.
There was a tape jammed in the drive, so the tape robot was refusing to load any new tapes, but kept on writing and restoring from the same tape over and over again.
Stupidly, we didn't do any 'check a tape from 3 weeks ago' for a while.
Lost quite a bit of data. We still have the md5sums though... Still gets shivers thinking about it