Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Content-defined chunking added to Bazel

buildbuddy.io

Post reply on HN
Print view View original

1–10 of 11 posts

Re: Content-defined chunking added to Bazel

#4

Post by a_t48 »

This is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.

Re: Content-defined chunking added to Bazel

#5

Post by rienbdj »

londons_explore wrote: post #2

Doesn't this mean that malicious inputs can deliberately cause super tiny or super huge chunks?

Bazel caches tend to have a size limit.

You need to trust your build execution machine anyway. They have your source code and you will be executing the artifacts that they produce!

Re: Content-defined chunking added to Bazel

#6

Post by dotwaffle »

I have done extensive research on CDC and it almost never works out because most utilities don't create compressed archives in an "rsyncable" (rsync does CDC) format, I actually saved a lot of storage using restic when I switched my backups of certain things so that files were stored in archives uncompressed, and sorted in a stable order. I know syncthing eventually removed CDC and just went with constant-size block sizes.

Bazel, on the other hand, is completely in control of this, and it makes perfect sense to do this at that point -- and it seems to be a relatively efficient implementation too, really nice to see!

Re: Content-defined chunking added to Bazel

#7

Post by stabbles »

a_t48 wrote: post #4

This is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.

Isn't that rather difficult given the `.tar.gz` layers?

Re: Content-defined chunking added to Bazel

#8

Post by tracnar »

stabbles wrote: post #7
a_t48 wrote: post #4

This is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.

Isn't that rather difficult given the `.tar.gz` layers?

It also supports .tar but that's probably not very commonly used.

Re: Content-defined chunking added to Bazel

#9

Post by auscompgeek »

stabbles wrote: post #7
a_t48 wrote: post #4

This is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.

Isn't that rather difficult given the `.tar.gz` layers?

In theory eStargz layers should be amenable to CDC.

Re: Content-defined chunking added to Bazel

#10

Post by a_t48 »

stabbles wrote: post #7
a_t48 wrote: post #4

This is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.

Isn't that rather difficult given the `.tar.gz` layers?

I have a custom pull client/registry/builder that uses a different format, but can output standard OCI if needed.
Post reply on HN