Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CircleCI Down

status.circleci.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: CircleCI Down

#41

Post by brightball »

ShakataGaNai wrote: post #28
brightball wrote: post #22

Earlier quoted context omitted.

I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several…

So they did the thing they were recommended but didn't take some further steps, on this one issue. Clearly that means they are totally incompetent? Even though the people dealing with DMARC issues are probably IT & Marketing, not the DevOps & Engineering people who are running the product.

A p=none record is barely different from not having a record at all...and yes at this point a tech company without an enforced record is a major red flag. It's been a decade since the standard went public, it's required at the federal level already and in many EU countries it's being mandated for businesses in general.

Most 3rd party senders today already insist that you setup DKIM as part of your setup process and if that happens, you're going to pass a DMARC check. It's hard to setup for older companies with thousands of servers in their own data centers that are each individually sending email. Cloud native companies sending their email through a few 3rd parties like Sendgrid/Postmark or a newsletter tool are EASY to setup.

I'm mentioning this on a post about their infrastructure being down for 6 hours because yes, it's related. Email delivery for the primary domain is absolutely an IT, Engineering, Operations and Security problem, not a marketing problem. It goes directly to the application especially when one of the main facets of the application is to send emails about your repos and login credentials.

Blame shifting it to the marketing department does not hold up.

When multiple people are commenting on this post about just how frequently their outages are happening it shows a problem in the overall infrastructure mindset for it to continue. Maybe they know exactly what the problem is and somebody higher up is keeping them from fixing it in order to prioritize other things.

Either way, for company that's supposed to be providing a core devops function to have outages that frequently as well as making it dead simple to spoof email that looks like it's coming straight from them...it's not a good look.

Re: CircleCI Down

#42

Post by brightball »

tedmiston wrote: post #31
brightball wrote: post #22

Earlier quoted context omitted.

I'm not all that surprised. A friend saw a phishing email that was imitating them because they lacked a DMARC record. Sent them explicit instructions on how to fix it by adding a DMARC policy and all they did was create a p=none record that doesn't prevent direct imitation. That's definitely the first step, but eventually you need to turn it up to p=quarantine for it to do you any good and it's been a while (several…

To be fair, DMARC quarantining is actually a pain in the ass and will likely break things for people outside of engineering or IT. In a growing or big company, there are always more and more legitimate emails from third-party senders added all the time. I agree that reviewing is the first step, but not everyone needs to take further steps. And I highly doubt CircleCI is unique here. I think it's a massive leap to con…

Everyone absolutely needs to take the next step. Without it, you're inviting direct phishing against your user base.

For an core devops tool, that's not okay.

Re: CircleCI Down

#43

Post by watermelon0 »

tedmiston wrote: post #36
nerdjon wrote: post #29

I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment) We get all the customization I want and it scales just fine. But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left…

I used CircleCI a long time ago, just before and after 2.0, and thought it was fine. These days GitHub Actions is awesome and full featured and does everything I want though without getting in the way. I doubt I'd use anything besides GHA unless I wanted to decouple from GitHub as a dependency. Even then, I wouldn't be surprised if someone else hasn't already come up with an open source or local runner for the GHA ya…

Two downsides of GHA that I could find:

- you are limited to 2 core / 7 GB instance, whereas CircleCI offers up to 16 cores / 64 GB (for example, building software usually scales proportionately to machine size, and this can in theory be up to 8x faster on CircleCI)

- no support for ARM instances

Re: CircleCI Down

#44

Post by folkrav »

macinjosh wrote: post #16
folkrav wrote: post #11

Earlier quoted context omitted.

I know what CircleCI is. Just wanted to understand what part of it you were referring to, which seems to be SaaS in general. I'm honestly not really convinced about self-hosting really avoiding untimely downtime, but whatever works for you and your team. E.g. I've worked in a business who self-host their Gitlab instance, there was a non-negligible amount of work for backups/upgrades on top of troubleshooting performa…

Every service will have down time. It is about the locus of control and agency. When I self host I can decide when to apply updates and do maintenance. When something does go wrong I can dive in, fix the problem, and move on with my day. Most people don't want any responsibility though so they do everything they can to push work off to a SaaS or cloud provider. That way when the SHTF they can browse reddit and point…

> Most people don't want any responsibility though so they do everything they can to push work off to a SaaS or cloud provider. That way when the SHTF they can browse reddit and point their boss to the status page.

Phew, those are a lot of assumptions. Many businesses choose cloud services cause they take the tradeoff between that cost or having someone at hand to maintain the infrastructure, depending on its complexity. I've worked with employers using many cloud providers for years now, the amount of times I would _legitimately_ be prevented from working at all by outages is extremely low.

The agency part, I do get. But come on.

Re: CircleCI Down

#45

Post by thinkindie »

to all those saying "why you don't spin a GitLab CI instance" - we are a small team, we want to focus on shipping code that adds value to our customers, not maintaining something that has been largely commoditised.

Re: CircleCI Down

#46

Post by z00b »

a1445c8b wrote: post #18
ismayilzadan wrote: post #10

Earlier quoted context omitted.

You are totally right, catastrophes happen and I also wish that they get past this with as little stress as possible. The whole reason of my assumption was the lack of description in their updates for the incident that is going on for 6 hours. Maybe little more detail would give me a hint that everything is under control, but I didn't feel that when I read their updates.

It’s probably just because they had the choice of either focusing all their energy on fixing the problem asap or setting aside some of it to write a more detailed description that’s also fit for public consumption. Given the severity, they probably chose the former since whatever descriptive, reassuring description they put out there isn’t going to be actionable anyway.

Hi folks. As the CircleCI CTO, I appreciate your patience here and all the feedback. It's true that we are focused on getting customers moving again over sharing more detailed information, but will aim to do better in providing a bit more in our updates. status.circleci.com provides real time updates for both how we're tackling outages and more detailed incident reports. We will post more information there about this incident once we are on the other side and have comprehensive detail.

Re: CircleCI Down

#47

Post by softfalcon »

Nextgrid wrote: post #25

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

Folks are likely going to say they can't do this "at scale". However, I've been in both the "small shop with Jenkins running on a Dell blade" and also in the "mega-corp with lots of devs everywhere using all kinds of CI pipelines". I think there is merit in what you're saying, but I suggest an addendum.

In both cases, have seen the "CI is down" issues crop up.

There seems to be a middle ground that I have yet to fully do, but have seen folks do and heard it works well:

Make it so folks can deploy their CI pipeline anywhere. In my case, I would create Docker containers that can run the CI pipeline, pipe the output/result to anywhere you want and have it be able to run anywhere.

The goal is that you could deploy it on a VPS somewhere, or on a Dell blade in your closet, or even run it on a dev's machine. The idea is that you want to reliably be able to run the CI steps anywhere without jank while following the full process the same as if it was running on your dedicated job runner solution. Redundancy, if you will.

I think the idea isn't necessarily "Jenkins in your company closet is better" and more of "how easily can you setup your CI pipeline to get your work done?"

Re: CircleCI Down

#48

Post by xyst »

Nextgrid wrote: post #25

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

in some cases that single server is bob's 2016 MBP because management is too cheap to buy a dedicated server

Re: CircleCI Down

#49

Post by rhodysurf »

tauntz wrote: post #39
nerdjon wrote: post #29

I still fail to see the heavily opinionated appeal of CircleCI over running a dockerized Jenkins instance (and agents) in AWS. (Or GitHub Actions or any other managed CI environment) We get all the customization I want and it scales just fine. But I am also still annoyed that when CircleCI announced their templates, they did not offer you the ability to have private templates (or something along those lines, it left…

There are _a lot_ of companies/teams that don't need/want to manage the complexity that comes with setting up and configuring generic CI environments. They want to get going as fast as possible and never have to think about what's going on behind the scenes - that's especially true in some specific areas. For example mobile apps. 90% of people aren't too happy about managing all the complexity that comes with having…

This is exactly why we use it. We legit don’t have time to manage more in house services and need CI. Eventually we probably move to GitHub actions but that isn’t free for orgs.

Re: CircleCI Down

#50

Post by CSSer »

justin_oaks wrote: post #30
Nextgrid wrote: post #25

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

I agree with the sentiment that people should evaluate whether or not they need an external service to run their builds. That said, there are a number of reasons to not use the Jenkins server in the office: 1) Someone on staff needs to maintain it. 2) A single hardware failure can cause significant downtime. 3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact depl…

Not to change the subject but point one is such a sticky point that a lot of the non-technicals or semi-technicals get overly hung up on in my experience. It's a very valid concern, and personally in this case I would say even more so because the potential pitfalls of maintaining physical hardware are no joke. However, I keep running into situations where that point is touted out for even compromising, intermediary alternatives like setting up your own build pipeline on a cloud provider or even using a cloud provider instead of a niche SaaS solution at all.

Forgive me if this is just "one of those problems" everyone has to deal with. I'm admittedly still pretty young and naive, and I've found myself in a mentorship/leadership adjacent role at a fast growing company where my relatively small team has a vastly deeper well of technical knowledge and aptitude than the rest of the organization. So I do wonder sometimes if I'm wrong about it, but my intuition, and it seems that of GP's too it seems, drives me to spring to this kind of solution (that we should be doing whatever it is ourselves because it's mission critical for what we do) often and it really gets to me that it's not so readily heard.

It's somewhat ironic because a big part of the point of HN in general is building and promoting these kind of progressive services, but the longer I'm in the industry and the more advanced my knowledge and responsibilities get, the more I keep finding that it's really hard to communicate the gaps in between the marketing claims these SaaS tools promote to higher-ups who don't see the incurred technical debt and wasted time working around their shortcomings. I've found this to be especially true when the issues are ones that one only bumps up against after rigorous use or problems that decision makers try to address by encouraging the addition of yet another third party tool to the mix.

Post reply on HN