Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CircleCI Down

status.circleci.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: CircleCI Down

#31

Post by tedmiston »

brightball wrote: post #22
oxff wrote: post #2

> Update - We are investigating multiple possible causes, including database changes and code changes. Sounds like they haven't got the first clue about what is causing it.

I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several…

To be fair, DMARC quarantining is actually a pain in the ass and will likely break things for people outside of engineering or IT. In a growing or big company, there are always more and more legitimate emails from third-party senders added all the time.

I agree that reviewing is the first step, but not everyone needs to take further steps. And I highly doubt CircleCI is unique here. I think it's a massive leap to conclude "lack of confidence in taking their infrastructure seriously" from not knowing the reason why they haven't flipped the switch from none to reject or quarantine.

Technically sophisticated users know that email spoofing is already rampant and to watch for signs of it in their email client. I'm not saying it's not a good idea, but that flipping the switch is not that simple and comes with significant downsides in a company with many services and users.

IMO I think going to the next level with DMARC is usually more of a prioritization or cost-benefit analysis type decision than a competence once.

Re: CircleCI Down

#33

Post by hotpotamus »

tapoxi wrote: post #17
rorymalcolm wrote: post #13

Earlier quoted context omitted.

Hosting any significantly scaled CI/CD system is very difficult work and since every team in your organisation usually touches it you often have a large blast radius. I've seen downtimes take a long time to recover from even when self hosted, compliance is also hard. It's a good candidate for SaaS IMO.

I've been the owner of our ~90 user GitLab deploy and its been mostly painless over the past two years, we have it installed on an autoscaling GKE cluster subscribed to the 'stable' channel. I helm upgrade it monthly.

You have a strange way of saying "self-hosted" ;) But it sounds like you've found a happy medium. This is a subject I may need to look into the in relatively near future.

Re: CircleCI Down

#34

Post by XCSme »

brightball wrote: post #22
oxff wrote: post #2

> Update - We are investigating multiple possible causes, including database changes and code changes. Sounds like they haven't got the first clue about what is causing it.

I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several…

Just linking the short guide I wrote (mostly for myself) to help with email auth stuff: https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...

Re: CircleCI Down

#35

Post by egwor »

justin_oaks wrote: post #30
Nextgrid wrote: post #25

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

I agree with the sentiment that people should evaluate whether or not they need an external service to run their builds. That said, there are a number of reasons to not use the Jenkins server in the office: 1) Someone on staff needs to maintain it. 2) A single hardware failure can cause significant downtime. 3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact depl…

And also consider the cost of running it there too (heat/energy/noise/space costs)

Re: CircleCI Down

#36

Post by tedmiston »

nerdjon wrote: post #29

I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment) We get all the customization I want and it scales just fine. But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left…

I used CircleCI a long time ago, just before and after 2.0, and thought it was fine.

These days GitHub Actions is awesome and full featured and does everything I want though without getting in the way. I doubt I'd use anything besides GHA unless I wanted to decouple from GitHub as a dependency.

Even then, I wouldn't be surprised if someone else hasn't already come up with an open source or local runner for the GHA yaml files as a stopgap.

Re: CircleCI Down

#38

Post by XCSme »

Nextgrid wrote: post #25

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

I personally really dislike remote building. It's so much faster to do everything locally, even more if you can do it on your own machine (machine which is likely to be very powerful, close to the state-of-the art performance, as the tech geeks we are).

Re: CircleCI Down

#39

Post by tauntz »

nerdjon wrote: post #29

I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment) We get all the customization I want and it scales just fine. But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left…

There are _a lot_ of companies/teams that don't need/want to manage the complexity that comes with setting up and configuring generic CI environments. They want to get going as fast as possible and never have to think about what's going on behind the scenes - that's especially true in some specific areas. For example mobile apps. 90% of people aren't too happy about managing all the complexity that comes with having to juggle Xcode versions or Android emulators etc in a scalable way.

..at least that's my experience and why I founded an extremely opinionated CI service for mobile apps years ago :)

Re: CircleCI Down

#40

Post by x86_64Ubuntu »

carimura wrote: post #23
frays wrote: post #20

This was the final straw. I'm switching over to Buildkite now.

I'm switching to X and will be happy until either, a) X gets large enough to matter and also has scale issues. b) X gets acquired by Y and sends a "what a great journey we're so excited that X will have Y's resources" email and then inevitably becomes just another forgotten tool under Y /s sorta

>...inevitably becomes just another forgotten tool under Y

Not before Y cuts all development to X. It doesn't always happen, but if you see Embarcadero buying a piece of your tech stack, find an alternative, immediately.

Post reply on HN