Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

source.dev

Post reply on HN
Print view View original

11–20 of 66 posts

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#11

Post by zokier »

Ericson2314 wrote: post #5

The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?

Well they also claim to be able to cache build steps somehow build-system independently.

> As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused

> SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration.

Which sounds way too good to be true.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#12

Post by bityard »

Meh, content marketing for a commercial biz. There are no interesting technical details here.

I was a build engineer in a previous life. Not for Android apps, but some of the low-effort, high-value tricks I used involved:

* Do your building in a tmpfs if you have the spare RAM and your build (or parts of it) can fit there.

* Don't copy around large files if you can use symlinks, hardlinks, or reflinks instead.

* If you don't care about crash resiliency during the build phase (and you normally should not, each build should be done in a brand-new pristine reproducible environment that can be thrown away), save useless I/O via libeatmydata and similar tools.

* Cross-compilers are much faster than emulation for a native compiler, but there is a greater chance of missing some crucial piece of configuration and silently ending up with a broken artifact. Choose wisely.

The high-value high-effort parts are ruthlessly optimizing your build system and caching intermediate build artifacts that rarely change.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#14

Post by everlier »

vzaliva wrote: post #13

It sounds from the page that it is Android-source-code specific. Why? Could this work with any source code base?

If my understanding is correct, this only makes sense for codebases that do not fit in memory of a largest build box an organisation can run

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#15

Post by ongy »

jonnrb wrote: post #8
ongy wrote: post #4

While it looks like at least some of the team are ex-googlers, this isn't the srcfs we know from piper (Google internal tools). Looks like it's similar in some ways. But they also don't tell too much and even the self-hosting variant is "Talk to us" pricing :/

WDYM this seems very familiar. At commit deadbeef I don't need to materialize the full tree to build some subcomponent of the monorepo. Did I miss something? And as for pricing... are there really that many people working on O(billion) lines of code that can't afford $TalkToUs? I'd reckon that Linux is the biggest source of hobbyist commits and that checks out on my laptop OK (though I'll admit I don't really do much…

Oh yea, this is "srcfs the idea" but not "srcfs the project".

I.e. this isn't something battel tested for hundreds of thousands of developers 24/7 over the last years. But a simple commercial product sold by people that liked what they used.

Well, since android is their flagship example, anyone that wants to build custom android releases for some reason. With the way things are, you don't need billions of code of your own code to maybe benefit from tools that handle billions of lines of code.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#17

Post by forrestthewoods »

The world desperately needs a good open source VFS that supports Windows, macOS, and Linux. Waaaaay too many companies have independently reinvented this wheel. Someone just needs to do it once, open source it, and then we can all move on.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#18

Post by 7e »

bityard wrote: post #12

Meh, content marketing for a commercial biz. There are no interesting technical details here. I was a build engineer in a previous life. Not for Android apps, but some of the low-effort, high-value tricks I used involved: * Do your building in a tmpfs if you have the spare RAM and your build (or parts of it) can fit there. * Don't copy around large files if you can use symlinks, hardlinks, or reflinks instead. * If y…

That’s all basic stuff, and none of it solves what this product claims to.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#19

Post by fukka42 »

zokier wrote: post #11
Ericson2314 wrote: post #5

The headline times are a bit ridiculous. Are they trying to turn https://github.com/facebook/sapling/blob/main/eden/fs/docs/O... or some git fuse thing into a product?

Well they also claim to be able to cache build steps somehow build-system independently. > As the build runs, any step that exactly matches a prior record is skipped and the results are automatically reused > SourceFS delivers the performance gains of modern build systems like Bazel or Buck2 – while also accelerating checkouts – all without requiring any migration. Which sounds way too good to be true.

Seems viable if you can wrap each build stap with a start/stop signal.

At the start snapshot the filesystem. Record all files read & written during the step.

Then when this step runs again with the same inputs you can apply the diff from last time.

Some magic to automatically hook into processes and doing this automatically seems possible.

Re: SourceFS: A 2h+ Android build becomes a 15m task with a virtual filesystem

#20

Post by 7e »

forrestthewoods wrote: post #17

The world desperately needs a good open source VFS that supports Windows, macOS, and Linux. Waaaaay too many companies have independently reinvented this wheel. Someone just needs to do it once, open source it, and then we can all move on.

This. Such a product also solves some AI problems by matting you version very large amounts of training data in a VCS like git, which can then be farmed out for distributed unit testing.
Post reply on HN