I want this but self hosted/integrated into our CI (Gitlab in our case).
Please fill in this form: https://www.source.dev/demo . We’re prioritizing cloud deployments but are keen to hear about your use case and see what we can do.
SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
41–50 of 66 posts
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#42While it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/
Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.
Doesn't perforce have a VFS that works on Windows?
I think it was made by Microsoft; https://github.com/microsoft/p4vfs
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#43Earlier quoted context omitted.
It’s definitely not ccache as they cover that under compiler wrapper. This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot). It’s unclear how they benchmark - they probably include checkout time of the codebase which artificially inflates the cost of the build (you only checkout once). It’s a virtual filesystem like what Facebook has open sourced…
Re: including checkout, it’s extremely unlikely. source: worked on Android for 7 years, 2 hr build time tracks to build time after checkout on 128 core AMD machine; checkout was O(hour), leaving only an hour for build if that was the case.
Obviously this is the best-case, hyper-optimized scenario and we were careful not to inflate the numbers.
The machine running SourceFS was a c4d-standard-16, and if I remember correctly, the results were very similar on an equivalent 8-vCPU setup.
As mentioned in the blog post, the results were 51 seconds for a full Android 16 checkout (repo init + repo sync) and ~15 minutes for a clean build (make) of the same codebase. Note that this run was mostly replay - over 99 % of the build steps were served from cache.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#44Earlier quoted context omitted.
I imagine it would be as magic as blaze vs bazel out in the wild. That is, you need still someone(s) to do a ton of hard work to make it work right but when it does you do get the magic.
You’re absolutely right - SrcFS and EdenFS were inspirations for SourceFS. The challenge with those systems is that they’re tightly coupled with the tools, infrastructure, and even developer distros used internally at Google and Meta, which makes them hard to generalize. SourceFS aims to bring that “Piper-like” experience to teams outside Google - but in a way that works with plain Git, Repo, and standard Linux envir…
It's a shame that AI is ruining certain phrases, the "You’re absolutely right" was appropriate but I've been trained reading so many AI responses to roll my eyes at that.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#45Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#46Meh, content marketing for a commercial biz. There are no interesting technical details here. I was a build engineer in a previous life. Not for Android apps, but some of the low-effort, high-value tricks I used involved: * Do your building in a tmpfs if you have the spare RAM and your build (or parts of it) can fit there. * Don't copy around large files if you can use symlinks, hardlinks, or reflinks instead. * If y…
That’s all basic stuff, and none of it solves what this product claims to.
We hear you on the “we want more technical blogs” part - they’ll be coming once we get a breather. We kept this first post high-level to reach a broader audience. Thanks for reading!
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#47Why tf does an electric vehicle need 500m+ lines of code
Some people actually write tests.
We actually picked a fairly conservative number - there are even larger automotive codebases today.
For example, Mercedes’ MB.OS: “is powered by more than 650 million lines of code” - see: https://www.linkedin.com/pulse/behind-scenes-mbos-developmen...
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#48Earlier quoted context omitted.
A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3 So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.
Didn't tup do something like that? https://gittup.org/tup/index.html Haven't looked at it in a while, no idea if it got adoption. But initially the article sounded like it was describing a mix of tup and Microsoft's git vfs ( https://github.com/microsoft/VFSForGit ) mushed together. But doing that by itself is probably a pile of work already.
Yes, you are correct - SourceFS also caches and replays build steps in a generic way.
It works surprisingly well, to the point where it’s hard to believe until you actually see it in action (here is a short demo video, but it probably isn't the best way to showcase it: https://youtu.be/NwBGY9ZhuWc?t=76 ).
We intentionally kept the blog post light on implementation details - partly to make it accessible to a broader audience, and partly because we will be posting gradually some more details. Sounds like build caching/replay is high on the desired blogpost list - ack :-).
The build-system integration used here was a one-line change in the Android build tree. That said, you’re right - deeper integration with the build system could push the numbers even further, and that’s something we’re actively exploring.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#49Tldr: your build system is so f'd that you have gigs of unused source and hundreds of repeated executions of the same build step. They can fix that. Or, you could, I dunno, fix your build?
You could just have a mono-repo with a large amount of assets that aren't always relevant to pull.
Incremental builds and diff only pulls are not enough in a modern workflow. You either need to keep a fleet of warm builders or you need to store and sync the previous build state to fresh machines.
Games and I'm sure many other types of apps fall into this category of long builds, large assets, and lots of intermediate build files. You don't even need multiple apps in a repo to hit this problem. There's no simple off the shelf solution.
Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem
#50Tldr: your build system is so f'd that you have gigs of unused source and hundreds of repeated executions of the same build step. They can fix that. Or, you could, I dunno, fix your build?
You could just have a mono-repo with a large amount of assets that aren't always relevant to pull. Incremental builds and diff only pulls are not enough in a modern workflow. You either need to keep a fleet of warm builders or you need to store and sync the previous build state to fresh machines. Games and I'm sure many other types of apps fall into this category of long builds, large assets, and lots of intermediate…
For a large amount of sometimes-relevant assets, is mapping them over NFS a bad solution? SourceFS also gets them across the network on demand, right?
And a fleet of warm builders seems pretty reasonable at that scale.
SourceFS sounds useful for extra smart caching but some of these problems do sound like they're just bad fixable configuration.