CircleCI: Our First Postmortem

any reason why you guys choose to pipe the compressed content into `tar xzf` process inside the container instead of extracting it outside and overlay the extracted content onto the container via overlayfs or something similar ?

Re: CircleCI: Our First Postmortem

#6

Post by dlowe » Tue, May 21, 2013, 4:58 PM UTC

dqminh wrote: post #5

any reason why you guys choose to pipe the compressed content into `tar xzf` process inside the container instead of extracting it outside and overlay the extracted content onto the container via overlayfs or something similar ?

Piping it in allows the build driver to be agnostic about the physical location of the container.

Re: CircleCI: Our First Postmortem

#7

Post by dlowe » Tue, May 21, 2013, 5:13 PM UTC

chris_wot wrote: post #4

Troubleshooting can be a bitch. Could you add a tl;dr though?

I'm not sure if I can do any better than "troubleshooting can be hard", frankly. The actual details are all tangled together in a way that resists summary.

Re: CircleCI: Our First Postmortem

#8

Post by chris_wot » Wed, May 22, 2013, 8:29 AM UTC

dlowe wrote: post #7

chris_wot wrote: post #4

Troubleshooting can be a bitch. Could you add a tl;dr though?

I'm not sure if I can do any better than "troubleshooting can be hard", frankly. The actual details are all tangled together in a way that resists summary.

Just tell customers that there were queue backlogs caused by slow git clones that were exacerbated by server failures that occurred due to kernel panics and LVM snapshot problems. These were resolved, but due to MTU configuration changes made during troubleshooting there were further outages; later on an unrelated bug in schejulur caused another outage.

However, all these issues are now resolved and your service is far more robust because of it.