Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CircleCI Down

status.circleci.com

Post reply on HN
Print view View original

11–20 of 60 posts

Re: CircleCI Down

#11

Post by folkrav »

capableweb wrote: post #9
folkrav wrote: post #5

Earlier quoted context omitted.

> Not a fan of these types of services myself What do you mean by "these types of services"?

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

I know what CircleCI is. Just wanted to understand what part of it you were referring to, which seems to be SaaS in general. I'm honestly not really convinced about self-hosting really avoiding untimely downtime, but whatever works for you and your team.

E.g. I've worked in a business who self-host their Gitlab instance, there was a non-negligible amount of work for backups/upgrades on top of troubleshooting performance issues once in a while, amongst other things in the same vein.

Re: CircleCI Down

#12

Post by capableweb »

folkrav wrote: post #11
capableweb wrote: post #9

Earlier quoted context omitted.

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

I know what CircleCI is. Just wanted to understand what part of it you were referring to, which seems to be SaaS in general. I'm honestly not really convinced about self-hosting really avoiding untimely downtime, but whatever works for you and your team. E.g. I've worked in a business who self-host their Gitlab instance, there was a non-negligible amount of work for backups/upgrades on top of troubleshooting performa…

The part about "untimely downtime" is about that CircleCI decides themselves when to push updates (which is the most common reason services has downtime), instead of you deciding when to upgrade/push updates. If you have a big migration/change coming up, you'd put pause on upgrading the CI/CD service as you don't want to muck with it while pushing out other organization-wide changes.

Granted, self-hosting comes with it's own share of problems too, no solution is a silver-bullet without any issues, but being able to "freeze" things to a stable mode helps to stabilize other processes.

Re: CircleCI Down

#13

Post by rorymalcolm »

capableweb wrote: post #9
folkrav wrote: post #5

Earlier quoted context omitted.

> Not a fan of these types of services myself What do you mean by "these types of services"?

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

Hosting any significantly scaled CI/CD system is very difficult work and since every team in your organisation usually touches it you often have a large blast radius. I've seen downtimes take a long time to recover from even when self hosted, compliance is also hard. It's a good candidate for SaaS IMO.

Re: CircleCI Down

#14

Post by stepanhruda »

capableweb wrote: post #9
folkrav wrote: post #5

Earlier quoted context omitted.

> Not a fan of these types of services myself What do you mean by "these types of services"?

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

They offer to self-host by the way.

Re: CircleCI Down

#15

Post by capableweb »

rorymalcolm wrote: post #13
capableweb wrote: post #9

Earlier quoted context omitted.

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

Hosting any significantly scaled CI/CD system is very difficult work and since every team in your organisation usually touches it you often have a large blast radius. I've seen downtimes take a long time to recover from even when self hosted, compliance is also hard. It's a good candidate for SaaS IMO.

As mentioned in a sibling comment: Yes, self-hosting comes with it's own problems, there are no silver-bullets in this industry.

However, I'd still argue that choosing when downtime can happen is important when you're pushing out larger changes to larger organizations. You don't want to be in the middle of a borked migration when your CI/CD service craps out and you can't rollback/push more updates or even restore backups as it was all automated via your CI/CD service.

Re: CircleCI Down

#16

Post by macinjosh »

folkrav wrote: post #11
capableweb wrote: post #9

Earlier quoted context omitted.

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

I know what CircleCI is. Just wanted to understand what part of it you were referring to, which seems to be SaaS in general. I'm honestly not really convinced about self-hosting really avoiding untimely downtime, but whatever works for you and your team. E.g. I've worked in a business who self-host their Gitlab instance, there was a non-negligible amount of work for backups/upgrades on top of troubleshooting performa…

Every service will have down time. It is about the locus of control and agency. When I self host I can decide when to apply updates and do maintenance. When something does go wrong I can dive in, fix the problem, and move on with my day.

Most people don't want any responsibility though so they do everything they can to push work off to a SaaS or cloud provider. That way when the SHTF they can browse reddit and point their boss to the status page.

Re: CircleCI Down

#17

Post by tapoxi »

rorymalcolm wrote: post #13
capableweb wrote: post #9

Earlier quoted context omitted.

CircleCI is a CI/CD solution (ala SaaS) that you don't host yourself. Many (myself included) prefer to host mission-critical services ourselves to avoid untimely downtime.

Hosting any significantly scaled CI/CD system is very difficult work and since every team in your organisation usually touches it you often have a large blast radius. I've seen downtimes take a long time to recover from even when self hosted, compliance is also hard. It's a good candidate for SaaS IMO.

I've been the owner of our ~90 user GitLab deploy and its been mostly painless over the past two years, we have it installed on an autoscaling GKE cluster subscribed to the 'stable' channel. I helm upgrade it monthly.

Re: CircleCI Down

#18

Post by a1445c8b »

ismayilzadan wrote: post #10
onion2k wrote: post #6

Earlier quoted context omitted.

I don't think you can say that. They might know what happened, but it could still be hard to recover from. Catastrophes happen. If you deploy something that has a destructive migration you can't easily roll back without reverting to a backup and there's a problem that you've not seen in QA then you're in for a bad time. This is compounded if you also discover your backup process hasn't worked properly for a while. If…

You are totally right, catastrophes happen and I also wish that they get past this with as little stress as possible. The whole reason of my assumption was the lack of description in their updates for the incident that is going on for 6 hours. Maybe little more detail would give me a hint that everything is under control, but I didn't feel that when I read their updates.

It’s probably just because they had the choice of either focusing all their energy on fixing the problem asap or setting aside some of it to write a more detailed description that’s also fit for public consumption. Given the severity, they probably chose the former since whatever descriptive, reassuring description they put out there isn’t going to be actionable anyway.

Re: CircleCI Down

#19

Post by plumefar »

ismayilzadan wrote: post #10
onion2k wrote: post #6

Earlier quoted context omitted.

I don't think you can say that. They might know what happened, but it could still be hard to recover from. Catastrophes happen. If you deploy something that has a destructive migration you can't easily roll back without reverting to a backup and there's a problem that you've not seen in QA then you're in for a bad time. This is compounded if you also discover your backup process hasn't worked properly for a while. If…

You are totally right, catastrophes happen and I also wish that they get past this with as little stress as possible. The whole reason of my assumption was the lack of description in their updates for the incident that is going on for 6 hours. Maybe little more detail would give me a hint that everything is under control, but I didn't feel that when I read their updates.

When facing such large scale issues, communicating properly is very hard: Several teams might be investigating several possible root causes in parallel, and you might change your mind over time as to what is the most probable root cause.

So you might end up communicating something ("we think it comes from X, we're fixing it that way"), just to find yourself changing you mind a few minutes later.

Changing your message is usually not well perceived, even though that's actually normal during an investigation.

I would not like to be in charge of the communication. Finding the balance between saying too much or too little is tricky.

Post reply on HN