Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Google Cloud Container Builder

cloudplatform.googleblog.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Introducing Google Cloud Container Builder

#61

Post by arghimonmobile0 »

kmicklas wrote: post #50
mickronome wrote: post #48

Earlier quoted context omitted.

I like what I understand to be the core ideas of NixOS to be, but i have to say: There might be a chance that some century, people both in the industry and outside understand that new syntax, or languages, are rarely necessary, or even beneficial. Use different semantics if you must, but please use some existing syntax for your DSL. I'm so tired of learning a pointless new variety of almost the same thing, only with…

Agreed, Nix the language is probably unnecessary. (The main benefit I can think of is avoiding depending on the moving target that is basically any other scripting language they could have tried to co-opt.)

I feel like I should plug Guix at this point. Pretty much similar to Nix, but the config language is Guile, which is Scheme.

Re: Introducing Google Cloud Container Builder

#62

Post by skj »

lobster_johnson wrote: post #53
skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

This looks great. One question: We're currently using Drone (the open-source self-hosted version), which allows us to define a number of parallel sidecar containers ("compose") as part of the build pipeline. We start things like PostgreSQL and Elasticsearch like this because a lot of tests need to exercise the actual data storage instead of (or in addition to) mocking/emulating them. The container builder thus double…

My general advice is to do integration testing after push.

You say that you value the testing harness having the same set of dependencies as your deployable artifact. With GCCB, you are not limited to one image with your build - it can be any number of them.

So, something like

  1. build the deployable as gcr.io/project/service:${REVISION_ID}
  2. build the testing harness as gcr.io/project/testing:${REVISION_ID}
  3. build the db container as gcr.io/project/db:${REVISION_ID}
Then, you can spin up a test environment using the ${REVISION_ID} tags, and do your integration testing there with the same nice guarantees.

That all said, it's certainly possible to do fancier things within the build itself. Using the gcr.io/cloud-builders/docker build step, you can run any docker command. We haven't seen the need to directly support a docker-compose build step, but making it would be easy (different entrypoint on the docker build step) and it would automatically connect to the worker host's daemon.

You can also run several of these docker build steps in parallel, or run one that does 'docker run --name=foo -d ...' and have the others run with '--link foo:foo', etc.

We try not to limit the sorts of things you can do, and provide bonus features like the docker daemon's socket, service account credentials etc.

You don't even need to build a container image, if you don't want to. Just provide some container images and we'll run them for you, pipe the logs back to you, and let you know how it went.

Re: Introducing Google Cloud Container Builder

#63

Post by skj »

lobster_johnson wrote: post #55
skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

How are secrets intended to be managed? E.g.: NPM auth tokens (for private NPM modules), Github tokens (e.g. for private Ruby gems or submodules), Slack webhooks, etc. I suppose one could bake that stuff into a base image, or store it in a Kubernetes secret that the build service account has access to.

We currently don't have direct support for secrets other than your service account's access tokens. We hope to support more arbitrary secrets in the future.

As you said, you can bake things into a base image (ew) or use your credentials to fetch them from somewhere else (I suggest Cloud Storage).

Re: Introducing Google Cloud Container Builder

#64

Post by lobster_johnson »

skj wrote: post #62
lobster_johnson wrote: post #53

Earlier quoted context omitted.

This looks great. One question: We're currently using Drone (the open-source self-hosted version), which allows us to define a number of parallel sidecar containers ("compose") as part of the build pipeline. We start things like PostgreSQL and Elasticsearch like this because a lot of tests need to exercise the actual data storage instead of (or in addition to) mocking/emulating them. The container builder thus double…

My general advice is to do integration testing after push. You say that you value the testing harness having the same set of dependencies as your deployable artifact. With GCCB, you are not limited to one image with your build - it can be any number of them. So, something like 1. build the deployable as gcr.io/project/service:${REVISION_ID} 2. build the testing harness as gcr.io/project/testing:${REVISION_ID} 3. buil…

Hm, that's a bit much. In particular as various apps have a different setup (we're a microservice shop and have tons of them, although a minority needs external dependencies for tests like these). Splitting it up like that also generates a dependency, which adds a burden on an already complicated workflow. Drone just solves that so much better by keeping everything in a single, simple build file.

Re: Introducing Google Cloud Container Builder

#65

Post by Avaclon »

skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

Hi, How does this work with regards to build parallelism? For examples builds that make use of outside parallel services such as ElectricAccelerator or Incredibuild?

Re: Introducing Google Cloud Container Builder

#66

Post by pbiggar »

alpb wrote: post #57

Disclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy.…

Question: was this on regular CircleCI, or the v2 beta?

Re: Introducing Google Cloud Container Builder

#67

Post by lobster_johnson »

skj wrote: post #63
lobster_johnson wrote: post #55

Earlier quoted context omitted.

How are secrets intended to be managed? E.g.: NPM auth tokens (for private NPM modules), Github tokens (e.g. for private Ruby gems or submodules), Slack webhooks, etc. I suppose one could bake that stuff into a base image, or store it in a Kubernetes secret that the build service account has access to.

We currently don't have direct support for secrets other than your service account's access tokens. We hope to support more arbitrary secrets in the future. As you said, you can bake things into a base image (ew) or use your credentials to fetch them from somewhere else (I suggest Cloud Storage).

The problem with uploading them somewhere is that you introduce an implicit dependency that must be documented and dealt with as part of the whole devops universe — it's just one more thing on top of a heap of other things.

It's much nicer, and more secure (in terms of locking down just one thing and not having to manage multiple to be able to centralize secrets somewhere. Whereas keeping things like this close the project is of course more convenient for developers.

Drone has tried two different systems, neither of which were ideal, and is working on a third one. I believe the third one also involves encrypted secrets committed to the project repo.

Re: Introducing Google Cloud Container Builder

#68

Post by alpb »

ferologics wrote: post #58
alpb wrote: post #57

Disclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy.…

What size is the project? Trying to guesstimate if it's worth investigating using 3rd party vs. [google] cloud container builder - speed tradeoffs, convenience, friction etc. Cheers.

I am simply building a Python container with about ~15 dependencies installed via pip and some ~50 MB of static assets added to the container. The base image comes from Docker Hub (hosted on S3/AWS CF).

If the final destination of your Docker image is GCR, it is really fast. You're basically not waiting for your build box to start up. Based on my experience the build starts almost right away after you push to the source control system, whereas CI systems like TravisCI/CircleCI spend quite some time preparing the build env (i.e. container) and running some preliminary commands, collecting artifacts/logs etc. Google Container Builder is looks like it is designed to build containers (not to run tests) and it does that quite fast.

Re: Introducing Google Cloud Container Builder

#69

Post by philip1209 »

alpb wrote: post #57

Disclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy.…

How does it work for monorepos with multiple docker images?

Re: Introducing Google Cloud Container Builder

#70

Post by alpb »

pbiggar wrote: post #66
alpb wrote: post #57

Disclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy.…

Question: was this on regular CircleCI, or the v2 beta?

Regular Circle CI. I do not have access to the v2 beta. For comparison here is the same Dockerfile built with Circle CI, it took 3m30s: https://cl.ly/0U1c2Q1n2S20/Image%202017-03-06%20at%209.50.31... and Google Cloud Container Builder: https://cl.ly/3E1c3l1L2Q3o/Image%202017-03-06%20at%209.52.01... although Cloud Container Builder does not list how long each step it takes, it totals 1m10s.
Post reply on HN