Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Google Cloud Container Builder

cloudplatform.googleblog.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: Introducing Google Cloud Container Builder

#91

Post by skj »

Avaclon wrote: post #65
skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

Hi, How does this work with regards to build parallelism? For examples builds that make use of outside parallel services such as ElectricAccelerator or Incredibuild?

GCCB does inside-VM parallelism with build steps, but the service itself does not support fan-out to multiple machines.

I think of GCCB as a building block on which a fan-out product can be built on. In fact, here is some vaporware: https://github.com/skelterjohn/flargo (a 20% project that is mostly just some ideas right now) (I am skelterjohn).

Re: Introducing Google Cloud Container Builder

#92

Post by skj »

splitbrain wrote: post #76

Could I use this to build images and push them to Docker Hub? The auto build feature at Docker Hub is awfully slow and often fails. Could this be a good alternative or am I missing the point of what this is?

We don't currently have a supported way to inject Docker Hub credentials into your build, but it's a feature we have our eye on.

Re: Introducing Google Cloud Container Builder

#93

Post by 7ewis »

Haven't tested out GCP yet, but am familiar with AWS and have played around with Docker a little.

Am I correct in saying that all this does is effectively a:

>docker build >docker push

But builds it all server side, then stores it in Google's Container Registry?

Re: Introducing Google Cloud Container Builder

#94

Post by skj »

7ewis wrote: post #93

Haven't tested out GCP yet, but am familiar with AWS and have played around with Docker a little. Am I correct in saying that all this does is effectively a: >docker build >docker push But builds it all server side, then stores it in Google's Container Registry?

No, that's not all it does, though you can certainly use it to do that.

This service runs a series of build steps on your source. It's an arbitrary execution engine with container building dressing on top (UI, CLI, etc).

One of the build steps might do 'docker build'. In fact, most of the build steps that GCCB has executed do just that! But it's not limited. You can provide container images for custom build steps to do whatever you like in a repeatable, automated fashion.

Re: Introducing Google Cloud Container Builder

#95

Post by ravdovid »

lobster_johnson wrote: post #67
skj wrote: post #63

Earlier quoted context omitted.

We currently don't have direct support for secrets other than your service account's access tokens. We hope to support more arbitrary secrets in the future. As you said, you can bake things into a base image (ew) or use your credentials to fetch them from somewhere else (I suggest Cloud Storage).

The problem with uploading them somewhere is that you introduce an implicit dependency that must be documented and dealt with as part of the whole devops universe — it's just one more thing on top of a heap of other things. It's much nicer, and more secure (in terms of locking down just one thing and not having to manage multiple to be able to centralize secrets somewhere. Whereas keeping things like this close the p…

Thank you for the input -- we do recognize the importance of being able to manage secure inputs to your builds and are investigating good solutions for this.

Re: Introducing Google Cloud Container Builder

#96

Post by ravdovid »

tjenkinsqs wrote: post #51
skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

While some of our targets will be Google App Engine Flexible, some will be GAE standard. Can we use it with GAE standard as well?

While we are focused on building Docker containers, you are not required to produce a container in your build. Any other built artifacts will need to be copied to persistent storage in one of your build steps.

Re: Introducing Google Cloud Container Builder

#97

Post by TheGuyWhoCodes »

jacobparker wrote: post #45
nchuhoai wrote: post #34

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…

I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.) Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental build…

I tried to compile tensorflow recently, it uses Bazel as the build system. When compiling it downloads external dependencies to be used during the build. It failed to retrieve some dependencies and I had to restart the proceeds only to fail on other dependencies and it seem to download all of them again (even though it was successful in downloading the other dependencies).

Googling for this issue I found out that if you fail and try you might even be blocked (from the servers) for trying to fetch too many dependencies and the solution was to "try later"...

Re: Introducing Google Cloud Container Builder

#98

Post by shostack »

skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

I'm not quite a novice but not a real developer by any means. I'm interested in learning more about containers and using them with my RoR side projects.

Is this new product a good place to start that would be relatively pain free for someone new to containers?

Re: Introducing Google Cloud Container Builder

#99

Post by ithkuil »

dottedmag wrote: post #78
kmicklas wrote: post #50

Earlier quoted context omitted.

Agreed, Nix the language is probably unnecessary. (The main benefit I can think of is avoiding depending on the moving target that is basically any other scripting language they could have tried to co-opt.)

To be fair, Nix language is completely different from other languages out there -- it employs only immutable data structures -- so using another language would not be possible. It does not excuse Nix to have such a braindamaged^W esoteric syntax.

Jsonnet is the closest I could find.

Re: Introducing Google Cloud Container Builder

#100

Post by skj »

shostack wrote: post #98
skj wrote: post #11

Hi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.

I'm not quite a novice but not a real developer by any means. I'm interested in learning more about containers and using them with my RoR side projects. Is this new product a good place to start that would be relatively pain free for someone new to containers?

If you're new to containers, I recommend starting by learning all about Dockerfiles. You can build them on your own machine, or with GCCB (using the `gcloud` command-line tool).

Dockerfiles make it challenging to keep your build-time environment separate from the run-time environment. For instance, you need the JDK to build your java app, but only the much smaller JRE to run it. Or maybe you want to use bazel.io for building, and have a completely different base image in mind for your deployment. Or maybe you want to bring in a unit testing harness and not publish the image unless all the tests pass.

Once these issues start to matter, GCCB has a great story. In essence, GCCB runs a series of containers with your source mounted in. So, you can run one step to run unit tests, another to build a binary, and a third to package everything into a container image.

GCCB also will build as many containers as you want in a single execution, so if you have several microservices that are all versioned together, you can build them at the same time and give them a `:${REVISION_ID}` tag - useful for production rollouts.

Post reply on HN