Live data from Hacker News

Please – A cross-language build system

please.build

211–220 of 257 posts

Re: Please – A cross-language build system

#211

Earlier quoted context omitted.

> You manually specified the tool to be run ($CC) and all of the arguments to that tool Notably, I didn't specify the arguments to be used to compile the source files. But if you insist: linklibrary = $(CC) -shared -o $1 $2 libfoo.so: $(OBJ) $(call linklibrary,$@,$(OBJ)) Notably, linklibrary can be defined according the platform, or according to dynamically set variables, giving you the same level of flexibility as m…

> ‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument. Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.

I don't follow.

Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language.

Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.

Re: Please – A cross-language build system

#212

Earlier quoted context omitted.

> You manually specified the tool to be run ($CC) and all of the arguments to that tool Notably, I didn't specify the arguments to be used to compile the source files. But if you insist: linklibrary = $(CC) -shared -o $1 $2 libfoo.so: $(OBJ) $(call linklibrary,$@,$(OBJ)) Notably, linklibrary can be defined according the platform, or according to dynamically set variables, giving you the same level of flexibility as m…

> Then you specify that the source files depend on the makefile. Then you gratuitously rebuild everything whenever the Makefile changes, even if you only changed a comment. Also this scheme is incorrect if the Makefile was written to allow command-line overrides of variables like CFLAGS, as many Makefiles do. But these are just details. The larger point is this. The language of Bazel is defined such that builds are a…

And now we are in agreement :)

Re: Please – A cross-language build system

#213

I can imagine a project that successfully replicates the entire google developer environment (distributed build system w/ caching, monorepo with presubmit checks, code review, testing, etc) would be successful since ex-googlers would be likely to advocate for it inside their own organizations and most orgs don't have the engineering time to build all this themselves. Without this tooling large organizations tend to s…

I agree. A big missing link at my organization is a VCS that can make it easier to checkout and operate on only a subset of a large (many GBs) monorepo.

Unfortunately, microsoft had to move away from VFS for Git to some a more limited-in-scope approach: https://github.com/microsoft/scalar

Re: Please – A cross-language build system

#214

> If you're familiar with Blaze / Bazel, Buck or Pants you will probably find Please very familiar Yes, so why would I use Please over any of them? I've spent close to 10min reading and have no idea why this exists or why anyone would use it. It looks like Bazel with a different config format, in which case why wouldn't one just use Bazel?

Please predates Bazel IIRC. It just didn’t gain as much traction. Please seems considerably easier to operate, but it doesn’t enjoy Bazel’s rigorous usage (although I’ve only encountered bugs when trying to use Bazel to build Python 3 projects).

Bazel’s Python integration quality is a level down from its Java, Go, and C++ integrations. It’s getting better though.

Re: Please – A cross-language build system

#215
post #126

Wish there was a new generation high level build system like this or Bazel or something but with decent JavaScript/node_modules support.

The Bazel and JS story is in its infancy. Things will get better once certain big companies have time to adopt and mature into it.

Re: Please – A cross-language build system

#216

Earlier quoted context omitted.

> The Makefile abstraction is "run this command when the output files don't exist or are older than the input files." You manually specify every tool to be run and all of its arguments. > The BUILD file abstraction is something like "I am declaring a C library with these sources and headers." This is wrong. Even ninja has generic rules. Here's an example of a minimal makefile: OBJ = src/a.o src/b.o src/c.o libfoo.so:…

> Here's an example of a minimal makefile: Your example does not contradict what I wrote. You manually specified the tool to be run ($CC) and all of the arguments to that tool. It's true that there is a level of indirection through the $CC variable, but you're still operating at the level of specifying a tool's command-line. > There's no reason this shouldn't be possible with make; it just hasn't been implemented so.…

FUSE, strace, and namespacing were all the mechanisms I found. Bazel uses separate wrapper program which you could reuse in other build systems, so there is no fundamental problem with adding a "hermetic builds" feature to other build systems like Meson or Cmake.

http://beza1e1.tuxen.de/hermetic_builds.html

Re: Please – A cross-language build system

#217

Earlier quoted context omitted.

> ‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument. Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.

I don't follow. Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language. Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.

Fundamentally, bazel and make treat "targets" differently. A make target is an invokable thing. That's about the extent of it. You have a dag of invokables, and invoking one will cause you to invoke all of its dependencies (usually, other people have discussed the limitations of make's caching already).

But let's look at how a rule is implemented in bazel[0]. Here's a rule "implementation" for a simple executable rule[1]:

    def _impl(ctx):
        # The list of arguments we pass to the script.
        args = [ctx.outputs.out.path] + [f.path for f in ctx.files.chunks]

        # Action to call the script.
        # actions.run will call "executable" with 
        # "arguments", saving the result to "output"
        # access to files not listed in "inputs" will
        # cause errors.
        ctx.actions.run(
            inputs = ctx.files.chunks,
            outputs = [ctx.outputs.out],
            arguments = args,
            progress_message = "Merging into %s" % ctx.outputs.out.short_path,
            executable = ctx.executable.merge_tool,
        )

    concat = rule(
        implementation = _impl,
        attrs = {
            "chunks": attr.label_list(allow_files = True),
            "out": attr.output(mandatory = True),
            "merge_tool": attr.label(
                executable = True,
                cfg = "exec",
                allow_files = True,
                default = Label("//actions_run:merge"),
            ),
        },
    )
This is, admittedly, not easy to follow at first glance. Concat defines a "rule" (just like cc_binary) that takes three arguments: "chunks", "out", and "merge_tool" (and "name", because every target needs a name).

Targets of this form have metadata, they have input and output files that are known and can be queried as part of the dag. Other types of rules can be tagged as test or executable, so that `blaze test` and `blaze run` can autodiscover test and executable targets. This metadata can also be used by other rules[2], so that a lot of static analysis can be done as a part of the dag creation, without even building the binary. To give an example, a rule like

    does_not_depend_on(
       name = "check_deps",
       src = ":opensource_thing",
       forbidden_deps = [
           "//super/secret:sauce",
       ]
    )
can be built and implemented natively within bazel by analyzing the dependency graph, so this test could actually run and fail before any code is compiled (in practice there are lots of more useful, although less straightforward to explain, uses for this kind of feature).

Potentially, one could create shadow rules that do all of these things, but you'd need to do very, very silly things like, off the top of my head, creating a shadow filesystem that keeps a file-per-make-target that can be used to query for dependency information (make suggests something similar for per-file dependencies[3], but bazel allows for much more complex querying). That's what I mean by "object-oriented". Targets in bazel and similar are more than just an executable statement with file dependencies. They're complex, user-defined structs.

This object-oriented nature is also what allows querying (blaze query/cquery/aquery), which are often quite useful for various sort of things like dead or unusued code detection or refactoring (you can reverse dependency query a library that defines an API, see all direct users and then be sure that they have all migrated to a new version). My personal favorite from some work I did over the past year or so was is `query --output=build record_rule_instantiation_callstack`, which provides a stacktrace of any intermediate startlark macros. Very useful when tracking down macros that conditionally set flags, but you don't know why, and a level of introspection, transparency, and debugability that just isn't feasible in make.

That's what I mean by object-oriented vs. text oriented. Bazel has structs with metadata and abstractions and functions that can be composed along and provide shared, well known interfaces. Make has text substitution and files. While a sufficiently motivated individual could probably come up with something in make that approximates many of the features bazel natively provides, I'm confident they couldn't provide all of them, and I'm confident it wouldn't be pretty or ergonomic.

[0]: https://docs.bazel.build/versions/master/skylark/rules.html

[1]: https://github.com/bazelbuild/examples/blob/master/rules/act...

[2]: https://docs.bazel.build/versions/master/skylark/aspects.htm...

[3]: https://www.gnu.org/software/make/manual/html_node/Automatic...

Re: Please – A cross-language build system

#218
post #166

Earlier quoted context omitted.

Make does support multiple outputs, though the syntax sucks. Most of what you are annoyed with though is like being annoyed at C for the same reasons: Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want... now, does it already come with whatever you want? No. I can appreciate wanting something which does. But such systems usually then give you wha…

> Make does support multiple outputs, though the syntax sucks. No, it doesn’t. There’s NO syntax for multiple outputs. If you can show me what the syntax is and prove me wrong, I’d love to see it. At best, there are workarounds for the lack of multiple output support, with various tradeoffs. > Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want...…

GNU Make does support multiple outputs, but the feature is very new (it's in the latest release that came out earlier this year), so if you didn't happen to catch that release announcement you probably missed it. The support is called 'grouped targets', documented in the second half of this page: https://www.gnu.org/software/make/manual/html_node/Multiple-... -- the syntax has &: in the rule line.

(One point you don't mention in your list of reasons why Make is successful is that it's reliably available everywhere. For projects that ship as source for others to build, that matters, and it delays uptake of fancy new build systems in that segment.)

Re: Please – A cross-language build system

#219

How big does the project have to be in order for this system to have benefits over Make?

The problem with Make is that it doesn't know what it's building, so it can't do anything smart. For incremental builds to work at all, you have to supply the smarts. Having worked on a number of make-based projects (I am most scarred by buildroot), I can tell you that people make mistakes with build rules. The project then devolves to doing a clean build for every change, turning what could be a few milliseconds of…

I am personally not convinced that any build system can be correct and general, but perhaps that’s my lack of experience speaking.

On that 30 minute note though: so, how big does the project need to be in order for Make not to be enough? And at that size, why wouldn’t the project invest the extra week it takes to get the Makefile correct?

Re: Please – A cross-language build system

#220
post #89

Earlier quoted context omitted.

Make build configurations can be difficult to understand for newcomers in the industry. If the goal is to obscure the code, by all means continue using the older tools. If the goal is continued maintenance, then encouraging new engineers to explore and read the codebase with tools they can comprehend is critical. disclaimer: have not used these particular tools, but the domain is nice and polite

I have use all of these tools and your take isn't accurate at all. Make doesn't obfuscate anything more than Plz or CMake or whatever. Here are some real reasons why Make isn't the end-all: - Make doesn't allow platform selection in a nice way. - Make doesn't work on Windows natively (no, NMake doesn't count). - Recursive make doesn't work well at all. - Make doesn't track byproducts or deleted artifacts. - Make does…

What tool would you pick over plz?
Post reply on HN