Live data from Hacker News

Tup – A file-based build system for Linux, OS X, and Windows

gittup.org

91–100 of 110 posts

Re: Tup – A file-based build system for Linux, OS X, and Windows

#91

Earlier quoted context omitted.

Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem: I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json). For the life of us, we could not…

Totally. Brocoli and Brunch are a bit better, but they all have the same fundamental problem: they require you to write plugins for things. The plugin model doesn't work here because writing plugins is so difficult, and you don't want to write this crap: you just want your project to build. In addition, all of the things you want to use in a build process expose a perfectly good shell interface, which probably stream…

Aside, I had a similar question when I learned PowerShell. It made me question even why I was writing so many script executables.

If we rely are devoted to using reusable and small components then a shell that can directly wrap our standard libraries (and has a type system rich enough to express maps and tree types) is indispensable to truly reusable software.

Then what a make system should be is a clear and smart way to express dependencies and workloads.

Re: Tup – A file-based build system for Linux, OS X, and Windows

#92

Earlier quoted context omitted.

Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem: I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json). For the life of us, we could not…

Totally. Brocoli and Brunch are a bit better, but they all have the same fundamental problem: they require you to write plugins for things. The plugin model doesn't work here because writing plugins is so difficult, and you don't want to write this crap: you just want your project to build. In addition, all of the things you want to use in a build process expose a perfectly good shell interface, which probably stream…

[deleted]

Re: Tup – A file-based build system for Linux, OS X, and Windows

#93

Earlier quoted context omitted.

Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem: I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json). For the life of us, we could not…

Totally. Brocoli and Brunch are a bit better, but they all have the same fundamental problem: they require you to write plugins for things. The plugin model doesn't work here because writing plugins is so difficult, and you don't want to write this crap: you just want your project to build. In addition, all of the things you want to use in a build process expose a perfectly good shell interface, which probably stream…

[deleted]

Re: Tup – A file-based build system for Linux, OS X, and Windows

#94

Earlier quoted context omitted.

Totally. Brocoli and Brunch are a bit better, but they all have the same fundamental problem: they require you to write plugins for things. The plugin model doesn't work here because writing plugins is so difficult, and you don't want to write this crap: you just want your project to build. In addition, all of the things you want to use in a build process expose a perfectly good shell interface, which probably stream…

Aside, I had a similar question when I learned PowerShell. It made me question even why I was writing so many script executables. If we rely are devoted to using reusable and small components then a shell that can directly wrap our standard libraries (and has a type system rich enough to express maps and tree types) is indispensable to truly reusable software. Then what a make system should be is a clear and smart wa…

I disagree, for so many reasons. PS is clunky and verbose, and textual data is actually so common that the unix toolkit is incredibly useful. Ad-hoc parsing comes up a lot, and if I have to parse a format, I'd rather it be text. Maps and trees aren't as important as you make them out to be. You can do key:value stores in text at O(n) cost, which is often G ood Enough, and rarely needed. And when you really need the extra power, take your pick: Python, Perl, Ruby, SCSH, and countless others.

A shell's primary job should be to support interactive use, and fast development of one-time-use scripts off the cuff, because those are what most of us do most commonly. PS doesn't support either all that well.

Re: Tup – A file-based build system for Linux, OS X, and Windows

#95
post #87

> tup, transitive verb: To have sex with. https://en.wiktionary.org/wiki/tup

It's an archaic britishism. The only time I've ever seen it actually used is in the works of Morgan Howell.

Shakespeare uses it in Othello, learnt something in 9th grade English!

Re: Tup – A file-based build system for Linux, OS X, and Windows

#96
post #23
post #16

Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These syste…

There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.

Ekam perhaps? https://github.com/sandstorm-io/ekam

Re: Tup – A file-based build system for Linux, OS X, and Windows

#97
post #47
post #23

Earlier quoted context omitted.

There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.

Well.. that's what tup does so it's probably what you're thinking of!

Yeah, tup does something similar with FUSE (afaik) but that's not what I'm thinking about.

Re: Tup – A file-based build system for Linux, OS X, and Windows

#98
post #39
post #13

Earlier quoted context omitted.

GNU make starts of with targets, and then talks the build graph (target -> source) to find source files which will re-build the targets. And GNU Make has to scan the files. Tup starts off with the list of sources which have changed, and then walks the build graph (source -> target) to find out which targets need to be rebuilt. And Tup gets notified of file changes. So where Make scans all targets and all source files…

GNU Make doesn't take exponential time, it takes ~linear time. The Y value on the graphs is exponential, because the input is growing exponentially. The graph is just rubbing in that a noop Tup build is O(1) instead of Make's O(n) by showing crazy large n. Which is certainly great on Tup's part![1] The Tup author's description of Tup's algorithm in comparison has always rubbed me the wrong way, though. It isn't some…

> Tup gets this list of changes by having a FUSE filesystem sit between the user and the actual filesystem, logging all changes

This is what I was really looking for. Instead of querying stat on all files at run time, it uses a filesystem to know at run time what changes were made.

Clever, so long as the FUSE layer never makes a mistake.

Re: Tup – A file-based build system for Linux, OS X, and Windows

#99

Earlier quoted context omitted.

Aside, I had a similar question when I learned PowerShell. It made me question even why I was writing so many script executables. If we rely are devoted to using reusable and small components then a shell that can directly wrap our standard libraries (and has a type system rich enough to express maps and tree types) is indispensable to truly reusable software. Then what a make system should be is a clear and smart wa…

I disagree, for so many reasons. PS is clunky and verbose, and textual data is actually so common that the unix toolkit is incredibly useful. Ad-hoc parsing comes up a lot, and if I have to parse a format, I'd rather it be text. Maps and trees aren't as important as you make them out to be. You can do key:value stores in text at O(n) cost, which is often G ood Enough, and rarely needed. And when you really need the e…

> Ad-hoc parsing comes up a lot, and if I have to parse a format, I'd rather it be text.

People often offer that "plain text is superior..." but neglect to recognize that PowerShell's model actually offers as superset of the current functionality of Bash, and that when object streams are required they are utterly indispensable.

It's also a uniquely eurocentric conceit that text is a simple stream of bytes. At the very least, it is a stream of bytes with an encoding label. Unicode is everywhere, and for some people is required to express their language. You can no more safely assume ISO-Latin-1 as you can UTF-8 or UTF-16.

Text isn't as simple as it used to be when we didn't give a damn about half the world's population.

> Maps and trees aren't as important as you make them out to be. You can do key:value stores in text at O(n) cost, which is often G ood Enough, and rarely needed.

We live in a world where Curling JSON objects or (even more problematically) protobuffs objects is the de-facto way we interact with the vast majority of remote services. Feel free to try and fit every workflow you have into jq. Feel free to keep working in an environment where even functional return values are not allowed and you keep repeating the same commands over and over.

> And when you really need the extra power, take your pick: Python, Perl, Ruby, SCSH, and countless others.

Quick question though: how is it this any different from the plugin approach from gulp? :)

> A shell's primary job should be to support interactive use, and fast development of one-time-use scripts off the cuff, because those are what most of us do most commonly. PS doesn't support either all that well.

Besides the awkward subquotation escapes for passing to commands, what exactly are you saying it can't do? Verbosity isn't exactly its problem, its editors are competent, its commandlets are quite capable, it's trivial to author more (and in many ways faster, e.g., adding a cmdlet with arguments is MUCH easier than the handrolled way in bash), you can directly call into a massive stdlib and any code you've written in .NET in any language that is supported by that target (including C#, F#, Nemerle, VB, Clojure and more) with native transitioning for all object types, and a bunch of other good features.

In essence that approach gives the shell a lot more room to do its work without calling out to black-box out-of-conceptual-model plugins, without sacrificing the power of being able to appeal to them when required.

Re: Tup – A file-based build system for Linux, OS X, and Windows

#100

Earlier quoted context omitted.

I disagree, for so many reasons. PS is clunky and verbose, and textual data is actually so common that the unix toolkit is incredibly useful. Ad-hoc parsing comes up a lot, and if I have to parse a format, I'd rather it be text. Maps and trees aren't as important as you make them out to be. You can do key:value stores in text at O(n) cost, which is often G ood Enough, and rarely needed. And when you really need the e…

> Ad-hoc parsing comes up a lot, and if I have to parse a format, I'd rather it be text. People often offer that "plain text is superior..." but neglect to recognize that PowerShell's model actually offers as superset of the current functionality of Bash, and that when object streams are required they are utterly indispensable. It's also a uniquely eurocentric conceit that text is a simple stream of bytes. At the ver…

>It's also a uniquely eurocentric conceit that text is a simple stream of bytes. At the very least, it is a stream of bytes with an encoding label. Unicode is everywhere, and for some people is required to express their language. You can no more safely assume ISO-Latin-1 as you can UTF-8 or UTF-16.

True, but how many of your text-processing utilities need to account for that? Most config file parsers will just continue to work if you insert UTF-8 into the stream, so long as you keep the markers it's working with the same.

>We live in a world where Curling JSON objects or (even more problematically) protobuffs objects is the de-facto way we interact with the vast majority of remote services. Feel free to try and fit every workflow you have into jq. Feel free to keep working in an environment where even functional return values are not allowed and you keep repeating the same commands over and over.

...I don't. Protobufs might have a market share in some places, but JSON has decent shell tooling, and you can often convert JSON to DSV in a pinch. I don't understand what you mean by "functional return values not allowed." Sure, EIAS is in full effect, but Shell pipelines are as functional as they come. And I don't repeat the same commands over and over. If I did, I'd give that set of commands a name.

>Quick question though: how is it this any different from the plugin approach from gulp? :)

Primary difference: The libraries are all already there, you don't have go get apt-get integration for the components you need, and it's often pretty quick to write what you need.

Verobosity is a huge problem. Most of the time with a shell, you'll be writing one-off scripts, or doing something interactive. If you were going to write something more than about 50 lines, or more complex than a couple of pipelines or an execution wrapper, you wouldn't be writing shell. PS is trying to go in a different direction, be a language for even those things that you'd go to another language for, but we've got perfectly adequate languages for that, tons of them, and it fails at things that shell is good at.

Post reply on HN