Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Concourse CI

concourse.ci

Post reply on HN
Print view View original

141–150 of 158 posts

Re: Concourse CI

#141

Post by jacques_chester »

jadeklerk wrote: post #132
zimbatm wrote: post #130

Avoid! Concourse looks good on the surface but it's really not that great. The UI is clunky. The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming. There are simple scenarios like deployment rollbacks who are hard to do. For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It's generally slow and w…

> Concourse looks good on the surface but it's really not that great. I've used teamcity, jenkins, gocd, circleci, concourse, and travisci. For multi-project systems, concourse is king. (I like travisci for by-itself, non-system projects) > The UI is clunky. What? You just said the UI looks good... It's simple and clean; everything is async javascript (no page loads). > The abstraction layer is too low and leads to a…

> Also I call rubbish on your 100% CPU. I have two workers t2.xlarge workers running 22 and 30 containers (like, right NOW) and neither is above 10% CPU (which, actually, I should make those a lot smaller). Don't run workers on a potato and you'll be fine.

I was on CF Buildpacks for a while, which is one of the earliest and more intensive users of Concourse.

Earlier versions of Concourse did stumble upon fairly hairy btrfs bugs that would enthusiastically choke CPUs to death if you had more than about 100 containers on a worker. This was particularly bad if, for any reason, you had a lot of jobs launching at once (I'm looking at you, 6-releases-simultaneously-NodeJS).

I can't remember if it was fixed by a kernel upgrade or whether they ditched btrfs. Either way it got fixed and I haven't seen it since.

Disclosure: I work for Pivotal.

Re: Concourse CI

#142

Post by jacques_chester »

fatterypt wrote: post #65
wlamartin wrote: post #61

Earlier quoted context omitted.

You can achieve this by clicking on a resource and hitting toggling the "power button" to off for each of the resources you don't want to include in the build.

While this may even work, it is not practical and it is a disappointing experience for a pipeline centric type of CI such as concourse. In other tools, hitting re-run would simply replay the job, with the same state it was executed in the first run. I would expect concourse to behave similarly, but no.

> In other tools, hitting re-run would simply replay the job, with the same state it was executed in the first run.

If the job failed, you get this. Otherwise I figure that the idea is that you're satisfied with those versions of the resources acted on by the job.

Disclosure: I work for Pivotal.

Re: Concourse CI

#143

Post by jacques_chester »

witten wrote: post #14

I took part in evaluating Concourse CI for the needs of my company (30+ devs). While it has amazing CI pipeline capabilities, we ultimately didn't select Concourse because it felt much more like a CI toolbox, requiring some development to put those tools to use. And what we really wanted was more of a turnkey CI product. Perhaps ironically, we ended up doing some development around the edges of the CI product we ulti…

> While it has amazing CI pipeline capabilities, we ultimately didn't select Concourse because it felt much more like a CI toolbox, requiring some development to put those tools to use.

This is actually a brilliant insight. I'd never thought of it that way. And I'm a one-eyed raving Concourse fan.

Re: Concourse CI

#144

Post by jacques_chester »

I'm late to the party.

Concourse is difficult to come to from other CI tools. A little more aloof. There have been real, serious implementation difficulties.

But I kinda love it, because it's a handful of simple ideas that unlock incredible power. It goes beyond "build and test each commit" to becoming a full project automation tool, a software manufacturing robot. When I talk to people about Concourse, I tell them: your pipeline and your tasks are production code. Keep the discipline, care and engineering practices that you bring to the apps and services you create.

The problem is that most of the deep tribal knowledge about how to get started and how to best apply it is locked up inside a handful of organisations, most notably Pivotal. I had been working on a video series which was meant to walk through both the concrete business of building pipelines, as well as the concepts of how best to do so.

Unfortunately visa conditions got in the way and I have abandoned that effort. Interested persons are welcome to email me for links to the first 4 episodes that I made, but be aware that it stops even more suddenly than Firefly.

Disclosure: I work for Pivotal.

Re: Concourse CI

#145

Post by jadeklerk »

mugsie wrote: post #136
jadeklerk wrote: post #133

Earlier quoted context omitted.

The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.

Well, to be clear, it's fly login --team-name internal --concourse-url " rel="nofollow">https:// -k (I think) Vs just going to the jenkins UI.

It's fly login -t

You can save targets.

Also, if you want logs to appear somewhere you can pretty easily set up a after-fail job to ftp em somewhere.

Re: Concourse CI

#146

Post by Ataraxic »

A little late to this discussion, but although I'm a big fan of many of the concepts in Concourse, I think it's lacking a lot of polish.

For example, I have actually implemented patterns within Jenkins to force all jobs to run inside of containers. A job is just a container + command with some linking using the jenkins pipeline plugin that reads some json configuration to determine how jobs are linked.

The primary issues I have surround the fact that my company uses kubernetes, thus we have no insight into the runc containers. Load balancing in concourse is non-existent. If a worker goes down due to load, if you bring it back up, it's going to go down immediately from all the jobs that have been triggered while the worker was offline. Not only that, the resource requirements seem pretty high. Recently a concourse worker stalled because the amount of volumes/images it was caching was over 100 gigs, and not knowing the internals, I wasn't sure what the best way was to clear this cache. Having to tell the infrastructure team that we just need to spend more money is a hard sell when we've upped the cpu, memory, storage, postgres disk, all more than once. I understand that different images have vastly different sizes, and jobs different amounts of work, but their need to be some clear suggestions for sizing. If there exist them now, I apologize but I haven't seen them.

So yeah I've had some fun developing in it, but more help making it reliable would be really nice. Also if kubernetes is absolutely the wrong way to run it, which it seems like, I'd have to be provided a better/easier alternative to really become an advocate.

Final note: has anyone actually setup metrics/monitoring for concourse that doesn't know BOSH? The docs describing it seem huge unless you already have the infrastructure pieces. Let's setup riemann (we already have statsd and no experience with riemann), emit to influxDB (we have prometheus, no experience with influxDB), then use Grafana (ok we already have that). I just wanted a better idea of disk, cpu, mem, # of containers, lifecycle without having to setup all these new pieces of infrastructure. Finally, just not that interested in BOSH, which all of the example metrics repo's are.

Re: Concourse CI

#147

Post by aarondl0 »

jadeklerk wrote: post #134
aarondl0 wrote: post #131

Excellently designed system. Very poorly implemented. Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines. The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibili…

> However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun. I've used concourse as a consumer for 3 years and I've very, very rarely seen any of the problems y…

UI freezes are completely client side and related to the elm implementation and your browser's execution of the code. The size of your VM doesn't matter at all.

ATC is a more of a dependency scheduler. The code [1] shows that it basically gets all pending jobs, and then runs them. There's no concept of queueing or maximum number of jobs, you just have to hope your limits are high enough (max containers, max tasks in systemd, max fds in the same) and that your machine doesn't fall over in the attempts.

The "massive" scheduling system also has no idea what nodes need work and which do not [2] so the idea is to heavily overprovision until it doesn't fall over (on top of already beefy requirements which others have alluded to in this thread).

You can not serialize multiple pipelines. Only within pipelines. If I have 10 pipelines, they will all run independently and there's nothing you can do about it other than attempt serialization with the pool resource (which we've recently had problems with - it also appears to be buggy and we're looking at submitting patches).

I've got a tremendous amount of experience with this system and I believe it's everything I made it out to be. The rebuttals you've provided to my issues are simply a lack of understanding of our context, not every user will have the same experience with any given product.

[1] https://github.com/concourse/atc/blob/master/scheduler/sched...

[2] https://github.com/concourse/concourse/issues/675

Re: Concourse CI

#148

Post by kieranajp »

EngineerBetter wrote: post #40
kieranajp wrote: post #31

We're using Concourse extensively at HelloFresh (>130 devs). It's not without its quirks, but I've little to complain about so far, except perhaps the polish of the UI.

Cool - did you folks know about the Concourse London User Group? The last two have been held just around the corner from you: https://www.meetup.com/Concourse-London-User-Group/

All our developers are Berlin based, so the London office isn't a great metric. Thanks anyway, though!

Re: Concourse CI

#149

Post by joncampbelldev »

mugsie wrote: post #80
joncampbelldev wrote: post #77

Earlier quoted context omitted.

For a lightweight pipeline based CI tool (with no docker requirements) you may like buildkite. I use it for almost everything I used to do in jenkins, although I still keep jenkins around as a glorified cron job / easy generic task runner. (not associated with buildkite, just a very happy customer)

buildkite looks nice - but on premise is a current requirement, as not all our repos are on the internet. Looks like a cool product though

buildkite can be done entirely on your own infrastructure (for the purposes of source control and builds), its an interesting setup:

- the ui is SaaS, but it doesn't care about your repos or build agents

- the build agent runs on your machines (they communicate outwards only with the SaaS product to decide if they should start building)

- all pipeline config is kept inside your repo

- you can hook up any source control to connect to buildkites webhooks, they just happen to have an integration to the common SaaS source control tools (github etc)

A nice extra is that running your own build agents lets you keep costs down. If you are on AWS they have a "one-click" setup for a cloud formation template that gives you an auto-scaling build environment

Re: Concourse CI

#150

Post by mugsie »

jacques_chester wrote: post #139
mugsie wrote: post #122

Earlier quoted context omitted.

Yeah - I can't remember the actual failure, but it did not work out well. We also need the VMs to survive across build steps. We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes

> We also need the VMs to survive across build steps. You might represent them as a pool resource, so they can be safely handed from job to job. A lot of teams do this for stuff where they want to share an expensive, long-lived, stateful resource. Disclosure: I work at Pivotal.

The VMs are per job - once the CI run tests that the orchestration has created the right system config on the VMs, and that basic integration tests work, we destroy the VMs.

I have worked in two companies now with Cloud Foundry distros, and both of them have started using Concourse, and then moved to Jenkins (for non Cloud Foundry projects - I shudder to think what would happen to a distro that attempted to not use Concourse)

Post reply on HN