CircleCI Down

> Update - We are investigating multiple possible causes, including database changes and code changes.

Sounds like they haven't got the first clue about what is causing it.

Lately, it's been down almost weekly. Not a fan of these types of services myself, but we do use it at work.

I don't think you can say that. They might know what happened, but it could still be hard to recover from.

Catastrophes happen. If you deploy something that has a destructive migration you can't easily roll back without reverting to a backup and there's a problem that you've not seen in QA then you're in for a bad time. This is compounded if you also discover your backup process hasn't worked properly for a while. If that happens you're facing some serious downtime, and the dilemma of either trying to fix the problem, or trying to rollback to the last working backup.

There's a good reason why grumpy old devs like me insist on writing docs, having playbooks, testing everything including non-code stuff, and we still fear major deploys. I have scars from exactly those sorts of disasters.

Hopefully the devs at Circle get past this with as little stress as possible, and they learn from what went wrong.

I wish it was only monthly. We have a Slack alert whenever there is reported downtime and it goes off at least once a week. Though granted it's seldom as business impacting as this current outage.

The problem CircleCI faces is that most hosted VCS now support CI/CD tools, as do most enterprise clouds. These will all have better integration with most peoples systems because they'll already be using the VCS or the public cloud (and if you're not using the public cloud you'd likely favor Jenkins / Concourse / etc over a cloud CI solution). So CircleCI's relevance is constantly being eaten at. The last thing they need it to damage their own reputation with these constant outages. I really hope CircleCI can turn around as it's great to see some competition but at this point in time I'm not feeling to optimistic about their long term future.

You are totally right, catastrophes happen and I also wish that they get past this with as little stress as possible. The whole reason of my assumption was the lack of description in their updates for the incident that is going on for 6 hours. Maybe little more detail would give me a hint that everything is under control, but I didn't feel that when I read their updates.