Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Concourse CI

concourse.ci

Post reply on HN
Print view View original

131–140 of 158 posts

Re: Concourse CI

#131

Post by aarondl0 »

Excellently designed system. Very poorly implemented.

Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines.

The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibility since there are no workers that get dirtier over time if you forget to clean up. The resource model is awesome. Very cool stuff. I'm hoping every CI system learns from what's here. Second to none in design.

However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun.

I really like Concourse and will hopefully one day be able to come back to it when its implementation is as solid as its paradigms are.

But I'd avoid use for now.

Re: Concourse CI

#132

Post by jadeklerk »

zimbatm wrote: post #130

Avoid! Concourse looks good on the surface but it's really not that great. The UI is clunky. The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming. There are simple scenarios like deployment rollbacks who are hard to do. For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and w…

> Concourse looks good on the surface but it's really not that great.

I've used teamcity, jenkins, gocd, circleci, concourse, and travisci. For multi-project systems, concourse is king. (I like travisci for by-itself, non-system projects)

> The UI is clunky.

What? You just said the UI looks good... It's simple and clean; everything is async javascript (no page loads).

> The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming.

Which is an intentional choice. If you don't like YAML, use one of the MANY yaml abstraction layers of your choice...

> There are simple scenarios like deployment rollbacks who are hard to do.

First of all, a CI system shouldn't be your answer to rollbacks. Your deployment system should handle that. Secondly, assuming your deployment system can do rollbacks, concourse has on-fail jobs that can trigger rollbacks just fine.

> For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and we had 100% CPU usage when the worker was doing almost nothing.

garden is used because cloudfoundry builds it. It is not slow... it is a light layer on top of runc (as opposed to docker which is a rather heavy layer on top of runc). You should pretty much never have to care about it, and in 3 years of using concourse I haven't had to - and we have some pretty gnarly large pipelines.

Also I call rubbish on your 100% CPU. I have two workers t2.xlarge workers running 22 and 30 containers (like, right NOW) and neither is above 10% CPU (which, actually, I should make those a lot smaller). Don't run workers on a potato and you'll be fine.

> Gitlab CI is much better. Or even Jenkins is better.

Ohhh you're trolling.

Re: Concourse CI

#133

Post by jadeklerk »

mugsie wrote: post #74
gabrielr wrote: post #71

Earlier quoted context omitted.

> Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible Have you tried `fly watch`? It will stream logs from in-flight builds, or show the full output of completed ones.

sure - I can use fly. if I got to the page, find the link to download the right version of fly, then remember the archine incantation to login, find a job, and find the logs. Or - I can go to my Jenkins build page, and click "Show full log". For me, I find the initial quick searching through logs much quicker in a browser. And, I have found that on longer builds the web UIs log streaming can actually crash the browse…

The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.

Re: Concourse CI

#134

Post by jadeklerk »

aarondl0 wrote: post #131

Excellently designed system. Very poorly implemented. Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines. The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibili…

> However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun.

I've used concourse as a consumer for 3 years and I've very, very rarely seen any of the problems you're describing, even on the older versions and certainly not in the last year or so.

> no scheduling to speak of

Concourse has a massive scheduling system built into it.. https://github.com/concourse/atc

Furthermore, you can configure jobs to run in serial (default is parallel).

> ui freezes

Put your `web` binary on a decently-sized VM and your problem should disappear. Also, don't have your workers on the same VM as your `web`.

Re: Concourse CI

#135

Post by jadeklerk »

krakensden wrote: post #113
henryaj wrote: post #62

Earlier quoted context omitted.

> UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs. Find this hard to believe - we have some pretty spectacular pipelines which render without a hiccup.

100+ jobs will break your browser. RabbitMQ has been complaining about it for a while.

100+ jobs in one pipeline?

Re: Concourse CI

#136

Post by mugsie »

jadeklerk wrote: post #133
mugsie wrote: post #74

Earlier quoted context omitted.

sure - I can use fly. if I got to the page, find the link to download the right version of fly, then remember the archine incantation to login, find a job, and find the logs. Or - I can go to my Jenkins build page, and click "Show full log". For me, I find the initial quick searching through logs much quicker in a browser. And, I have found that on longer builds the web UIs log streaming can actually crash the browse…

The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.

Well, to be clear, it's fly login --team-name internal --concourse-url " rel="nofollow">https:// -k (I think)

Vs just going to the jenkins UI.

Re: Concourse CI

#137

Post by joeseeder »

henryaj wrote: post #62
joeseeder wrote: post #44

Have been working with it for a few months, and I am not positive about it. UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs. Concept of teams is fine, but when you have to switch between them, even with oauth authentication, it is a pain. Job concurrency control is binary. We never have been able to have worker pool scale down without a hiccup, always some darn worke…

> UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs. Find this hard to believe - we have some pretty spectacular pipelines which render without a hiccup.

My experience after having the browser UI open is that intially stuff renders, then starts to take ages to update state ( these pretty pulsing frames are no longer showing or rendering badly ). Quite often switching to a different pipeline renders empty space.

Also, when you have around 100+ resources and jobs in one pipeline, in a vertical plane, they are impossible to identify

Re: Concourse CI

#138

Post by joeseeder »

jadeklerk wrote: post #135
krakensden wrote: post #113

Earlier quoted context omitted.

100+ jobs will break your browser. RabbitMQ has been complaining about it for a while.

100+ jobs in one pipeline?

Yes, some systems we work with have over 100 items ( jobs and resources) in one pipeline ... because microservices...

Re: Concourse CI

#139

Post by jacques_chester »

mugsie wrote: post #122
wlamartin wrote: post #88

Earlier quoted context omitted.

Booting VMs on your worker seems like exactly the kind of debris concourse is encouraging the avoidance of by enforcing the containerisation of your builds. That said, I can't think of a reason why you couldn't boot a VM in a privileged Garden container, have you tried?

Yeah - I can't remember the actual failure, but it did not work out well. We also need the VMs to survive across build steps. We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes

> We also need the VMs to survive across build steps.

You might represent them as a pool resource, so they can be safely handed from job to job. A lot of teams do this for stuff where they want to share an expensive, long-lived, stateful resource.

Disclosure: I work at Pivotal.

Re: Concourse CI

#140

Post by jacques_chester »

zimbatm wrote: post #130

Avoid! Concourse looks good on the surface but it's really not that great. The UI is clunky. The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming. There are simple scenarios like deployment rollbacks who are hard to do. For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and w…

> The UI is clunky.

The UI was originally designed with one end in mind: "Show me what is broken and take me there as fast as possible".

I and others have given lots of feedback about the strengths and limitations of that orientation, particularly trading off between a job-oriented and resource-version-oriented view of the world.

> There are simple scenarios like deployment rollbacks who are hard to do.

I've had this with the CF resource. The Concourse-y solution would be an undeploy resource.

> For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue.

The container engine is garden-runc, the actual containerising is the same code as Docker's. What I like a lot less is container orchestration. I have complained about this in a ha-ha-only-serious matter.

Disclosure: I work at Pivotal.

Post reply on HN