Devops Horror Stories

ch4ch4 wrote: post #3

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.

Not so much an example of why not to use budget providers, more an example of why you should build highly available infrastructure. I don't believe there is any provider, "budget" or not, that guarantees a servers reliability 100% of the time.

Re: Devops Horror Stories

#5

Post by ch4ch4 » Thu, Oct 31, 2013, 4:24 PM UTC

jamescun wrote: post #4

ch4ch4 wrote: post #3

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.

Not so much an example of why not to use budget providers, more an example of why you should build highly available infrastructure. I don't believe there is any provider, "budget" or not, that guarantees a servers reliability 100% of the time.

I was alluding more to the customer support aspect of it. If a technician spends one hour troubleshooting your network problems, then they've already lost their profit for the month.

Re: Devops Horror Stories

#6

Post by 699 » Thu, Oct 31, 2013, 4:34 PM UTC

little confused by their choices (simply 'move' to a different but similar cheap provider) why not diversify? I imagine they'd have fewer issues if they had a presence in multiple datacenters/providers

Re: Devops Horror Stories

#7

Post by perlpimp » Thu, Oct 31, 2013, 5:10 PM UTC

alt.sysadmin.recovery lives on! albeit in a web app. wonder if usenet is still alive...

Re: Devops Horror Stories

#8

Post by 0xbadcafebee » Thu, Oct 31, 2013, 5:14 PM UTC

My Devops horror stories, one sentence each:

- Somebody deployed new features on a Friday at 5pm.

- Fifteen hundred machines running mod_perl.

- Supporting Oracle - TWICE.

- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is.

- Troubleshooting a bug in a site, view source.... and see SQL in the JS.

Re: Devops Horror Stories

#9

Post by stevenklein » Thu, Oct 31, 2013, 5:15 PM UTC

0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

fun stuff

Re: Devops Horror Stories

#10

Post by danudey » Thu, Oct 31, 2013, 5:25 PM UTC

0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I built a system where our developers can do instant deployments of any of our software packages (and instant point-in-time rollbacks), and then do zero-downtime restarts of services.

Now we deploy dozens of times a day and I never get called on a Friday night because someone did something stupid.

Edit: I do get called when I did something stupid and it broke the deployment system. But that's gotten much rarer lately.