Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

source.dev

Post reply on HN
Print view View original

61–66 of 66 posts

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#63

Post by sudahtigabulan »

This reminds me of ClearCase and its MVFS.

Builds were audited by somehow intercepting things like open(2) and getenv(3) invoked by a compiler or similar tool, and each produced object had an associated record listing the full path to the tool that produced it, its accurate dependencies (exact versions), and environment variables that were actually used. Anything that could affect the reproducibility was captured.

If an object was about to be built with the exact same circumstances as those in an existing record, the old object was reused, or "winked-in", as they called it.

It also provided versioning at filesystem level, so one could write something like file.c@@/trunk/branch/subbranch/3 and use it with any program without having to run a VCS client. The version part of the "filename" was seen as regular subdirectories, so you could autocomplete it even with ancient shells (I used it on Solaris).

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#64

Post by bityard »

7e wrote: post #18
bityard wrote: post #12

Meh, content marketing for a commercial biz. There are no interesting technical details here. I was a build engineer in a previous life. Not for Android apps, but some of the low-effort, high-value tricks I used involved: * Do your building in a tmpfs if you have the spare RAM and your build (or parts of it) can fit there. * Don't copy around large files if you can use symlinks, hardlinks, or reflinks instead. * If y…

That’s all basic stuff, and none of it solves what this product claims to.

Not as basic as you seem to think.

I was brought into a team of 70-ish engineers working across 4-5 products. Big enterprise products written by very bright programmers. But build systems and infrastructure were not their core competency. Their flagship application took 6 hours to build when I was hired. I got it down to 30-45 minutes using a combination of the techniques above and a revamped build infrastructure. When I finally left that position, the build was much more modular, so you could rebuild a small part of it and glue it into a bunch of existing artifacts and have a final product in just a few minutes.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#65

Post by kevincox »

7e wrote: post #22
ongy wrote: post #4

While it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/

Google or Meta needs to open source their magic VFSes. Maybe Meta is closest with EdenFS.

I don't think it is very "magic".

The VFS gets you a few main benefits.

1. You can lazy download and checkout data as you need it. Assuming your build system is sufficiently realized to hide the latency this will save a lot of time as long as accessed data is significantly less then used data (say 2. You don't need to scan the filesystem to see what has changed. So things like commits, status checks and even builds checking for changes can target just what has actually changed.

Neither of these are particularly complex. The hardest part is integrating with the VCS and build system to take advantage of the change tracking. Git has some support for this (see fsmonitor) build I'm not aware of any build systems that do. (But you still get a lot of benefits without that.)

Post reply on HN