Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Google Cloud Container Builder

cloudplatform.googleblog.com

Post reply on HN
Print view View original

41–50 of 114 posts

Re: Introducing Google Cloud Container Builder

#41

Post by fishnchips »

kellyjandrews wrote: post #31
fishnchips wrote: post #16

Earlier quoted context omitted.

I like how Codeship approaches the problem - see here for inspiration https://documentation.codeship.com/pro/getting-started/steps... Still, they're super expensive and not very performant, so I welcome competition. What I'm missing here is an easy way to link a container being tested with external containers (eg DB).

Disclaimer: Codeship employee. Glad you like our approach. We are constantly working on performance, for sure. Since we offer dedicated build machines, there are always options for better performance. Appreciate the feedback, and feel free to reach out to me directly anytime if you have other questions or thoughts on how we can get better.

I did have a call some 3 months ago with your support rep and a sales person and provided extensive feedback - which was acknowledged and ignored. Which is precisely why I welcome new competitors that could wake you up from your comfy slumber ;)

Re: Introducing Google Cloud Container Builder

#42

Post by kentt »

gkop wrote: post #38
skj wrote: post #36

Earlier quoted context omitted.

One approach to this problem that I like is storing these dependencies in an intermediate image. Then, you can do an incremental build by referencing (rather than rebuilding) this intermediate image, and pay only the cost of downloading the raw data which should be cheap compared to the normal install process.

Why not leverage Docker's built-in image layer caching?

I think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems.

Yarn works the same.

Re: Introducing Google Cloud Container Builder

#43

Post by fishnchips »

skj wrote: post #19
fishnchips wrote: post #16

Earlier quoted context omitted.

I like how Codeship approaches the problem - see here for inspiration https://documentation.codeship.com/pro/getting-started/steps... Still, they're super expensive and not very performant, so I welcome competition. What I'm missing here is an easy way to link a container being tested with external containers (eg DB).

re: link the container with external containers, that's a bit tricky and generally I think that's a space you should use a "dev" environment for. We like to be the connection between source and registry, with your dev environment being after the registry.

That's not how I expect CI to work. I might be wrong though.

Re: Introducing Google Cloud Container Builder

#44

Post by kmicklas »

nchuhoai wrote: post #34

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…

Some century the industry will discover the NixOS model...

Re: Introducing Google Cloud Container Builder

#45

Post by jacobparker »

nchuhoai wrote: post #34

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…

I'd give this a look: https://github.com/GoogleCloudPlatform/cloud-builders/tree/m... (unfortunately key links to Bazel docs are 404'ing.)

Bazel is a general purpose build system that focuses on reproducibility (same build inputs -> same build outputs) and strict dependency specification (at various levels: file, subsystem, external package fetching etc.) enforced by sandboxing which allows for fast incremental builds. Bazel is the open-source version of Google's internal build-system.

More info here: https://bazel.build/

How external resources are handled: https://bazel.build/versions/master/docs/external.html

Obviously Bazel requires some buy-in which may be unappealing (at least for existing projects: migration/supporting multiple build systems.) There are, however, serious benefits (and a larger, but nascent, ecosystem of related open-source tooling.) The biggest gap at the moment is supported languages/etc.

(I don't work for Google, I just like Bazel.)

Re: Introducing Google Cloud Container Builder

#46

Post by jzelinskie »

nchuhoai wrote: post #34

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…

Full Disclosure: I work on the Quay team at CoreOS

The Quay build system has implemented caching. The builder preemptively calculates the hashes of Dockerfile commands and then sends them to an API endpoint that finds the tag for the most similar tree of command hashes in a given repository. This tag is pulled before ever attempting to build the Dockerfile.

Google's solution seems far more general than just building Dockerfiles. I'm glad they're pushing for more innovation in this space.

Re: Introducing Google Cloud Container Builder

#47

Post by morgante »

kentt wrote: post #42
gkop wrote: post #38

Earlier quoted context omitted.

Why not leverage Docker's built-in image layer caching?

I think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems. Yarn works the same.

Having a separate FROM image doesn't help that situation either, since you have to rebuild it.

Re: Introducing Google Cloud Container Builder

#48

Post by mickronome »

kmicklas wrote: post #44
nchuhoai wrote: post #34

I'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all th…

Some century the industry will discover the NixOS model...

I like what I understand to be the core ideas of NixOS to be, but i have to say: There might be a chance that some century, people both in the industry and outside understand that new syntax, or languages, are rarely necessary, or even beneficial.

Use different semantics if you must, but please use some existing syntax for your DSL.

I'm so tired of learning a pointless new variety of almost the same thing, only with slightly different syntax, maybe some awkward string quoting, and some rather pointless syntactic sugar.

I'm not saying NixOS is the worst offender, but innovating in (mostly) superficial syntax, or using obscure syntax is almost certainly not well spent time except for a few very narrow niches/contexts. Especially not for prospective users.

I get it, it's fun inventing both languages and tools, I love doing it myself!

However, and in general: Don't invent a language when you need a tool, and vice versa.

Re: Introducing Google Cloud Container Builder

#49

Post by puzzle »

gkop wrote: post #38
skj wrote: post #36

Earlier quoted context omitted.

One approach to this problem that I like is storing these dependencies in an intermediate image. Then, you can do an incremental build by referencing (rather than rebuilding) this intermediate image, and pay only the cost of downloading the raw data which should be cheap compared to the normal install process.

Why not leverage Docker's built-in image layer caching?

That might be less effective if you can't guarantee that all the builds end up always on the same node.

Re: Introducing Google Cloud Container Builder

#50

Post by kmicklas »

mickronome wrote: post #48
kmicklas wrote: post #44

Earlier quoted context omitted.

Some century the industry will discover the NixOS model...

I like what I understand to be the core ideas of NixOS to be, but i have to say: There might be a chance that some century, people both in the industry and outside understand that new syntax, or languages, are rarely necessary, or even beneficial. Use different semantics if you must, but please use some existing syntax for your DSL. I'm so tired of learning a pointless new variety of almost the same thing, only with…

Agreed, Nix the language is probably unnecessary. (The main benefit I can think of is avoiding depending on the moving target that is basically any other scripting language they could have tried to co-opt.)
Post reply on HN