It sounds from the page that it is Android-source-code specific. Why? Could this work with any source code base?
I think the page itself answers your question pretty well.
SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
21–30 of 66 posts
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#22While it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/
Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#23The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?
Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.
Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#24Their performance claims are quite a bit ahead of the distributed android build systems that I've used, I'm curious what the secret sauce is.
Is it going to be anything more than just a fancier ccache?
It’s definitely not ccache as they cover that under compiler wrapper. This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot). It’s unclear how they benchmark - they probably include checkout time of the codebase which artificially inflates the cost of the build (you only checkout once). It’s a virtual filesystem like what Facebook has open sourced although they claim to also do build caching without needing a dedicated build system that is aware of this and that part feels very novel
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#25While it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/
Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.
I have thought about this, but also wondered if it would be as magic without the highly paid team of fantastic SRE and maintainers, and the ridiculous amount of disk and compute available to them.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#26The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?
Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.
You could manage this with a deterministic vm, cf antithesis.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#27Earlier quoted context omitted.
Is it going to be anything more than just a fancier ccache?
It’s definitely not ccache as they cover that under compiler wrapper. This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot). It’s unclear how they benchmark - they probably include checkout time of the codebase which artificially inflates the cost of the build (you only checkout once). It’s a virtual filesystem like what Facebook has open sourced…
Re: including checkout, it’s extremely unlikely. source: worked on Android for 7 years, 2 hr build time tracks to build time after checkout on 128 core AMD machine; checkout was O(hour), leaving only an hour for build if that was the case.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#28Earlier quoted context omitted.
Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.
I have thought about this, but also wondered if it would be as magic without the highly paid team of fantastic SRE and maintainers, and the ridiculous amount of disk and compute available to them.
I imagine it would be as magic as blaze vs bazel out in the wild. That is, you need still someone(s) to do a ton of hard work to make it work right but when it does you do get the magic.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#29Earlier quoted context omitted.
Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.
Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.
A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3
So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#30I want this but self hosted/integrated into our CI (Gitlab in our case).