51–60 of 114 posts
Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.
Earlier quoted context omitted.
I think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems. Yarn works the same.
Having a separate FROM image doesn't help that situation either, since you have to rebuild it.
Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.
We start things like PostgreSQL and Elasticsearch like this because a lot of tests need to exercise the actual data storage instead of (or in addition to) mocking/emulating them. The container builder thus doubles as a CI server, since every build needs to be tested anyway, and tests need to go through very much the same steps (install dependencies, build source and so on). Tests benefit from being run on the exact same filesystem that the final code runs on, guaranteeing that dependencies are identical.
From what I can tell, such a setup is not supported by GCCB?
I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…
I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.) Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental build…
Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.
I suppose one could bake that stuff into a base image, or store it in a Kubernetes secret that the build service account has access to.
I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…
I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.) Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental build…
I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy. However it took me about an hour to build this Circle CI and I had to do some tasks like set up Service Accounts, copy keys around etc.
Google Container Builder eliminates this completely, you just connect your GitHub account, pick what you want to build (every commit, branch or tag) and specify the image name you want to build and forget about it forever.
It runs faster compared to free CI services I've used in the past (probably because image pulls are fast b/c Google Networking, but I'm not sure how much CPU is allocated for builds). Also if you're pushing to GCR, then it'll be faster to push images as well. So overall pull/build/push cycle is just fast. To compare, build for a very simple image I have (base image docker.io/library/python) takes 3m30s on CircleCI vs 1m10s in Google Cloud Container Builder. That's 3x faster.
Disclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy.…
Hah, interesting to note how much dogfooding happens at Google before they make these technologies available to outside customers. This is a really smart move on their part.
That's also what Amazon does, right?
Now, core services such as ec2 and S3 are dogfooded for sure, but it is more like amazon doesn't eat their own dogfood until after the product becomes big.