131–140 of 140 posts
Earlier quoted context omitted.
I believe at least some parts of the tooling were developed by an embedded team from Pivotal Labs.
Not that I am aware of. It's possible they contributed to some of our OSS efforts, but I don't believe anything used internally was developed in that manner.
Earlier quoted context omitted.
We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles…
I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built. I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way acros…
Earlier quoted context omitted.
I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built. I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way acros…
I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?
My first approach was as an individual, which was quite difficult, because it took me a while to work out the differences between jobs and tasks and how to lay it all out. Lots of copying and pasting from other pipelines I studied.
At this point pretty much every team in Pivotal's Cloud Foundry division is running a Concourse pipeline, including the one I belong to. What's been interesting is how each team is experimenting with patterns that Concourse makes possible.
For example, I and my peers are now turning various things into "executable documents" -- the terminology sucks at the moment. Think of all the stuff buried in READMEs and wikis and long-forgotten cron jobs. How to build that special docker image that you only update every few months. The database backup. Keeping the blue-green deploy codepath warm.
When we find another one of these, we now encode it into our pipeline. That way, if I need to find it, it's there. And if it goes bad, the pipeline definition points me at where to go looking for everything of interest.
Another pattern that is emerging for us is "enforce project invariants". For example, we have multiple repos, so multiple Gemfiles and Dockerfiles, all of them with ruby versions set. At the front of our pipeline there is a little gateway to check that these are all identical. If not, it prints a table of versions found, so again, I know instantly where to go look. We have various other little invariants that turn days of insane debugging into a few seconds of sanity-checking.
None of this is novel. What Concourse lets me do is hoist good design practices out of code and apply them to CI. The final feedback loop I rely on can itself now be checked in, broken apart according to SOLID principles and even (my colleagues in Buildpacks are pioneering this) unit tested. Some colleagues in London have designed a fully generic pipeline that can be configured at runtime to build any of the five products they manage.
tl;dr fuck yeah Concourse.
Earlier quoted context omitted.
Along a similar grain, I'm curious if anyone's used Joyent's infrastructure stack internally... I know they offer it, but not familiar enough with it. It seems Solaris containers as a base for docker containers is a better security model, but not sure what parts, if enough is open to implement without paying consultation from Joyent to get started even. It's definitely a compelling model. I am only slightly surprised…
HashiCorp also has a pretty complete solution called Atlas : https://www.hashicorp.com/ ( different from the Netflix Atlas http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ) Joyent's Triton looks cool: https://www.joyent.com/ as does CoreOS's Tectonic (built on top of k8s): https://tectonic.com/
Hadn't looked at Atlas... I changed jobs into a larger company, and don't really have to deal with the ops side ever. Though eagerly awaiting a blessed docker based solution to become available in the org... it'll mean that projects can be developed more readily without being limited to the Java based infrastructure in place now.
Earlier quoted context omitted.
> So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D > "We need to scan your hosts so we can ensure that you're installing security patches." > "We don't do that." > "You don't install security patches?!?" > "Yeah, well, you see..." If you use a tool like zypper-docker, you can create a new image quickly that applies jus…
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched so…
Earlier quoted context omitted.
> So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D > "We need to scan your hosts so we can ensure that you're installing security patches." > "We don't do that." > "You don't install security patches?!?" > "Yeah, well, you see..." If you use a tool like zypper-docker, you can create a new image quickly that applies jus…
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched so…
The point of zypper-docker is to quickly roll out security fixes with essentially no downtime. zypper-docker also allows you to quickly check the health of all of the images and containers on your servers, so you can figure out which ones need updating. Also, zypper-docker allows you to apply RPM patches (something that's quite crucial for enterprises).
So no, it's not pointless. In fact it solves a problem that not many people have been working on solving: updating containers.
Earlier quoted context omitted.
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched so…
Sometimes you don't want to update all of your dependencies (isn't that the whole point of Docker), you might want to just apply security updates. Just putting "FROM whatever:latest" is a bad idea, because you don't know whether the "latest" will suddenly break your image. The point of zypper-docker is to quickly roll out security fixes with essentially no downtime. zypper-docker also allows you to quickly check the…
How are you going to know if the latest version of the upstream image breaks your package if you don't try it? E whole point of continuous integration is that there are no surprises when it comes time to push to production.
If ":latest" breaks your image you had better know:
A) Immediately.
B) What went wrong ASAP.
The integrated testing (that you're supposed to use with Docker best practices) should reveal if there's a problem and if it doesn't you'll still catch it during QA testing of your image.
The fact that ":latest" breaks your image should be nothing more than a few minutes to days of troubleshooting. While you're doing that your existing production images will keep on chugging away.
The only rule that you must not break is that you have the next release of your image out the door, ready for production within 30 days. Why? Because that's the maximum (hard-coded) lifetime we (and you should too) allow images to stay running.
Earlier quoted context omitted.
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched so…
Sometimes you don't want to update all of your dependencies (isn't that the whole point of Docker), you might want to just apply security updates. Just putting "FROM whatever:latest" is a bad idea, because you don't know whether the "latest" will suddenly break your image. The point of zypper-docker is to quickly roll out security fixes with essentially no downtime. zypper-docker also allows you to quickly check the…
Earlier quoted context omitted.
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched so…
But then you risk having untold other packages move under your feet. This is what concepts like "unattended-upgrades" address.
Earlier quoted context omitted.
I've seen something similar happen, switched to https://www.go.cd/
GoCD is very difficult to version-control and the interface is, to put it politely, in need of some love. At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out. Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with C…