My monthly Circle CI downtime, yay.
In my experience, GitHub actions is equally flaky
CircleCI Down
21–30 of 60 posts
Re: CircleCI Down
#22> Update - We are investigating multiple possible causes, including database changes and code changes. Sounds like they haven't got the first clue about what is causing it.
I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several weeks). Shouldn't have needed a random user to point it out in the first place.
I just don't have a tremendous amount of confidence that they take their infrastructure seriously at this point.
Re: CircleCI Down
#23This was the final straw. I'm switching over to Buildkite now.
I'm switching to X and will be happy until either,
a) X gets large enough to matter and also has scale issues.
b) X gets acquired by Y and sends a "what a great journey we're so excited that X will have Y's resources" email and then inevitably becomes just another forgotten tool under Y
/s sorta
Re: CircleCI Down
#24This was the final straw. I'm switching over to Buildkite now.
Jenkins is the most flexible system I've worked with, but Buildkite has been great and just gets the job done reliably. Big fan.
Re: CircleCI Down
#25I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.
Re: CircleCI Down
#26This was the final straw. I'm switching over to Buildkite now.
We've been with Buildkite since 2016, it's been a great experience, I really like the tool, and their team is amazing!
Re: CircleCI Down
#27I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.
To operate properly at the scale of hundreds of engineers pushing changes, you absolutely need a cluster of machines and a team to operate it.
It is significantly cheaper and more efficient to pay a third party.
Re: CircleCI Down
#28> Update - We are investigating multiple possible causes, including database changes and code changes. Sounds like they haven't got the first clue about what is causing it.
I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several…
So they did the thing they were recommended but didn't take some further steps, on this one issue. Clearly that means they are totally incompetent? Even though the people dealing with DMARC issues are probably IT & Marketing, not the DevOps & Engineering people who are running the product.
Re: CircleCI Down
#29I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment)
We get all the customization I want and it scales just fine.
But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left a bad taste in my mouth and we moved to Jenkins a couple months later)
Re: CircleCI Down
#30I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.
I agree with the sentiment that people should evaluate whether or not they need an external service to run their builds.
That said, there are a number of reasons to not use the Jenkins server in the office:
1) Someone on staff needs to maintain it.
2) A single hardware failure can cause significant downtime.
3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact deployment
4) Having your server on-site may be considered a security risk.
I'm not saying that a server in the office is a bad idea, I'm just saying that each business needs to consider the advantages and disadvantages. I'm sure there are those who could get by just fine with a server in the office.