Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

source.dev

Post reply on HN
Print view View original

51–60 of 66 posts

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#51

Post by jayd16 »

Dylan16807 wrote: post #50
jayd16 wrote: post #49

Earlier quoted context omitted.

You could just have a mono-repo with a large amount of assets that aren't always relevant to pull. Incremental builds and diff only pulls are not enough in a modern workflow. You either need to keep a fleet of warm builders or you need to store and sync the previous build state to fresh machines. Games and I'm sure many other types of apps fall into this category of long builds, large assets, and lots of intermediate…

For a large amount of sometimes-relevant assets, is mapping them over NFS a bad solution? SourceFS also gets them across the network on demand, right? And a fleet of warm builders seems pretty reasonable at that scale. SourceFS sounds useful for extra smart caching but some of these problems do sound like they're just bad fixable configuration.

> And a fleet of warm builders seems pretty reasonable at that scale.

It's actually pretty hard. The more builders you have the older the workspace gets and scaling up or cycling machines causes the next builds to be super slow. Game engines end up making central intermediate asset caches like Unreal's UBA or Unity's Cache Server.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#52

Post by blaz0 »

jayd16 wrote: post #49
jeffrallen wrote: post #33

Tldr: your build system is so f'd that you have gigs of unused source and hundreds of repeated executions of the same build step. They can fix that. Or, you could, I dunno, fix your build?

You could just have a mono-repo with a large amount of assets that aren't always relevant to pull. Incremental builds and diff only pulls are not enough in a modern workflow. You either need to keep a fleet of warm builders or you need to store and sync the previous build state to fresh machines. Games and I'm sure many other types of apps fall into this category of long builds, large assets, and lots of intermediate…

Yes, games are a common case where your repo can be very large but building your code only requires access to a small subset of it.

For example, the League of Legends source repo is millions of files and hundreds of GB in size, because it includes things like game assets, vendored compiler toolchains for all of our target platforms, etc. But to compile the game code, only about 15,000 files and 600MB of data are needed from the repo.

That means 99% of the repo is not needed at all for building the code, and that is why we are seeing a lot of success using VFS-based tech like the one described in this blog. In this case, we built our own virtual filesystem for source code based on our existing content-defined patching tech (which we wrote about a while ago [1]). It's similar to Meta's EdenFS in that we built it on top of the ProjFS API on Windows and NFSv3 on macOS and Linux. We can mount a view into the multimillion-file repo in 3 seconds, and file data (which is compressed and deduplicated and served through a CDN) is downloaded transparently when a process requests it. We use a normal caching build system to actually run the build, in our case FASTBuild.

I recently timed it, and I can go from having nothing at all on disk to having locally built versions of the League of Legends game client and server in 20 seconds on a 32-core machine. This is with 100% cache hits, similar to the build timings mentioned in the article.

[1] https://technology.riotgames.com/news/supercharging-data-del...

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#53

Post by blaz0 »

dijit wrote: post #42
7e wrote: post #22

Earlier quoted context omitted.

Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.

Doesn't perforce have a VFS that works on Windows? I think it was made by Microsoft; https://github.com/microsoft/p4vfs

There is also an identically named VFS, this time from the Perforce company itself [1]

[1] https://help.perforce.com/helix-core/server-apps/p4vfs/curre...

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#54

Post by MarsIronPI »

> Fast builds are what truly makes a difference to developer productivity. With SourceFS builds complete over 9x faster on a regular developer machine. This sets a new standard as it enables developers to get their sword fighting time back and speeds-up the lengthy feedback loop on CI pipelines.

Objection! Long build times are better for sword-fighting time. The longer it takes, the more sword-fighting we have time for!

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#55

Post by ndesaulniers »

vlovich123 wrote: post #24
cogman10 wrote: post #16

Earlier quoted context omitted.

Is it going to be anything more than just a fancier ccache?

It’s definitely not ccache as they cover that under compiler wrapper. This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot). It’s unclear how they benchmark - they probably include checkout time of the codebase which artificially inflates the cost of the build (you only checkout once). It’s a virtual filesystem like what Facebook has open sourced…

> This works for Android because a good chunk of the tree is probably dead code for a single build (device drivers and whatnot)

Device drivers would exist in kernel sources, not the AOSP tree.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#56

Post by CJefferson »

zokier wrote: post #11
Ericson2314 wrote: post #5

The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?

Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.

I used to use a python program called ‘fabricate’ which did this. If you track every file a compiler opens, then id the same compiler is run with the same flags, and no input changed, you can just drop a cached copy of the outputs in place.

I’m actually disappointed this type of thing never caught on, it’s fairly easy on Linux to track every file a program accesses, so why do I need to write dependency lists?

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#57

Post by CJefferson »

serbancon wrote: post #37

Hey everyone. I’m Serban, co-founder of Source.dev. Thanks for the upvotes and thoughtful discussion. I’ll reply to as many comments as I can. Nothing means more to an early-stage team than seeing we’re building something people truly value - thanks from all of us at Source.dev!

While I’m sure it’s much more advanced, out of interest is this similar to the Python tool ‘fabricate’, which would use strace to track all files a program read, and wrote?

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#58

Post by vlovich123 »

serbancon wrote: post #43
refulgentis wrote: post #27

Earlier quoted context omitted.

Re: including checkout, it’s extremely unlikely. source: worked on Android for 7 years, 2 hr build time tracks to build time after checkout on 128 core AMD machine; checkout was O(hour), leaving only an hour for build if that was the case.

Obviously this is the best-case, hyper-optimized scenario and we were careful not to inflate the numbers. The machine running SourceFS was a c4d-standard-16, and if I remember correctly, the results were very similar on an equivalent 8-vCPU setup. As mentioned in the blog post, the results were 51 seconds for a full Android 16 checkout (repo init + repo sync) and ~15 minutes for a clean build (make) of the same codeb…

Do you have any technical blog post how the filesystem is intercepting and caching build steps? This seems like a non-obvious development. The blog alludes to a sandbox step which I’m assuming is for establishing the graph somehow but it’s not obvious to understand where the pitfalls are (eg what if I install some system library - does this interception recognize when system libraries or tools have changed, what if the build description changes slightly, how does the invalidation work etc). Basically, it’s a bold claim to be able to deliver Blaze-like features without requiring any changes to the build system.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#59

Post by ternus »

Mattwmaster58 wrote: post #44
serbancon wrote: post #38

Earlier quoted context omitted.

You’re absolutely right - SrcFS and EdenFS were inspirations for SourceFS. The challenge with those systems is that they’re tightly coupled with the tools, infrastructure, and even developer distros used internally at Google and Meta, which makes them hard to generalize. SourceFS aims to bring that “Piper-like” experience to teams outside Google - but in a way that works with plain Git, Repo, and standard Linux envir…

It's a shame that AI is ruining certain phrases, the "You’re absolutely right" was appropriate but I've been trained reading so many AI responses to roll my eyes at that.

The saving grace was that it was followed by a single hyphen, not an em-dash.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#60

Post by menaerus »

serbancon wrote: post #47
jeffbee wrote: post #9

Earlier quoted context omitted.

Some people actually write tests.

We actually picked a fairly conservative number - there are even larger automotive codebases today. For example, Mercedes’ MB.OS: “is powered by more than 650 million lines of code” - see: https://www.linkedin.com/pulse/behind-scenes-mbos-developmen...

650M LoC is certainly not a single codebase that you can "checkout" and "build". Also, the figure is a little bit hard to believe.
Post reply on HN