Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

source.dev

Post reply on HN
Print view View original

31–40 of 66 posts

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#31

Post by forrestthewoods »

7e wrote: post #20
forrestthewoods wrote: post #17

The world desperately needs a good open source VFS that supports Windows, macOS, and Linux. Waaaaay too many companies have independently reinvented this wheel. Someone just needs to do it once, open source it, and then we can all move on.

This. Such a product also solves some AI problems by matting you version very large amounts of training data in a VCS like git, which can then be farmed out for distributed unit testing.

HuggingFace bought XetHub which is really cool. It’s built for massive blobs of weight data. So it’s not a general purpose VCS VFS. The world still needs the latter.

I’d be pretty happy if Git died and it was replacing with a full Sapling implementation. Git is awful so that’d be great. Sigh.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#34

Post by bananaquant »

fukka42 wrote: post #19
zokier wrote: post #11

Earlier quoted context omitted.

Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.

Seems viable if you can wrap each build stap with a start/stop signal. At the start snapshot the filesystem. Record all files read & written during the step. Then when this step runs again with the same inputs you can apply the diff from last time. Some magic to automatically hook into processes and doing this automatically seems possible.

I think I got the magic part. You can store all build system binaries in the VFS itself. When any binary gets executed, VFS can return a small sham binary instead that just checks command line arguments, if they match, checks the inputs, and if they match, applies the previous output. If there is any mismatch, it can execute the original binary as usual and make the new output. Easy and no process hacking necessary.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#35

Post by vlovich123 »

rcxdude wrote: post #29
vlovich123 wrote: post #23

Earlier quoted context omitted.

Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.

A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3 So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.

Yeah that’s what I meant. I bet you the build must be invoked through a wrapper script that interposes all executables launched within the product tree. Complicated but I think it could work. Skipping steps correctly is the hard part but maybe you do that in terms of knowing somehow the files that will be accessed ahead of time by that processes and then skipping the launch and materializing the output (they also mention they have to run it once in a sandbox to detect the dependencies). But still, side effects in build systems seem difficult to account for correctly; I bet you that’s why it’s a “contact us” kind of product - there’s work needed to make sure it actually works on your project.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#36

Post by mook »

rcxdude wrote: post #29
vlovich123 wrote: post #23

Earlier quoted context omitted.

Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.

A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3 So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.

Didn't tup do something like that? https://gittup.org/tup/index.html Haven't looked at it in a while, no idea if it got adoption.

But initially the article sounded like it was describing a mix of tup and Microsoft's git vfs (https://github.com/microsoft/VFSForGit) mushed together. But doing that by itself is probably a pile of work already.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#37

Post by serbancon »

Hey everyone. I’m Serban, co-founder of Source.dev. Thanks for the upvotes and thoughtful discussion. I’ll reply to as many comments as I can. Nothing means more to an early-stage team than seeing we’re building something people truly value - thanks from all of us at Source.dev!

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#38

Post by serbancon »

fragmede wrote: post #28
mattnewton wrote: post #25

Earlier quoted context omitted.

I have thought about this, but also wondered if it would be as magic without the highly paid team of fantastic SRE and maintainers, and the ridiculous amount of disk and compute available to them.

I imagine it would be as magic as blaze vs bazel out in the wild. That is, you need still someone(s) to do a ton of hard work to make it work right but when it does you do get the magic.

You’re absolutely right - SrcFS and EdenFS were inspirations for SourceFS.

The challenge with those systems is that they’re tightly coupled with the tools, infrastructure, and even developer distros used internally at Google and Meta, which makes them hard to generalize. SourceFS aims to bring that “Piper-like” experience to teams outside Google - but in a way that works with plain Git, Repo, and standard Linux environments.

Also, if I’m not mistaken, neither SrcFS nor EdenFS directly accelerate builds - most of that speed comes from the build systems themselves (Blaze/Buck). SourceFS goes a step further by neatly and simply integrating with the build system and caching/replay pretty much any build step.

The Android example we’ve shown is just one application - it’s a domain we know well and one where the pain is obvious - but we built SourceFS in a way where we can easily integrate with a new build system and speed up other big codebases.

Also you’re spot on that this problem mostly affects big organizations with complex codebases. Here without the infrastructure and SRE support the magic does not work (e.g. think the Redis CVE 10.0 of last week or the AWS downtime of this week) - and hence the “talk to us”.

We plan to gradually share more interesting details about how SourceFS works. If there’s something specific you’d like us to cover - let us know - and help us crowd source our blogpost pipeline :-).

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#39

Post by fragmede »

serbancon wrote: post #38
fragmede wrote: post #28

Earlier quoted context omitted.

I imagine it would be as magic as blaze vs bazel out in the wild. That is, you need still someone(s) to do a ton of hard work to make it work right but when it does you do get the magic.

You’re absolutely right - SrcFS and EdenFS were inspirations for SourceFS. The challenge with those systems is that they’re tightly coupled with the tools, infrastructure, and even developer distros used internally at Google and Meta, which makes them hard to generalize. SourceFS aims to bring that “Piper-like” experience to teams outside Google - but in a way that works with plain Git, Repo, and standard Linux envir…

That “something specific” would be to invent some magic so you can get the advantages of the system without having an entire team to back it up!

Thank you for the thoughtful response!

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#40

Post by [deleted] »

rcxdude wrote: post #29
vlovich123 wrote: post #23

Earlier quoted context omitted.

Yeah, I agree. This part is hand waved away without any technical description of how they manage to pull this off since knowing what is even a build step and what dependencies and outputs are are only possible at the process level (to disambiguate multi threaded builds). And then there’s build steps that have side effects which come up a lot with CMake+ninja.

A fuse filesystem can get information about the thread performing the file access: https://man.openbsd.org/fuse_get_context.3 So they could in principle get a full list of dependencies of each build step. Though I'm not sure how they would skip those steps without having an interposer in the build system to shortcut it.

[deleted]
Post reply on HN