Show HN: Minimal build system using just /bin/sh • Hacker News Board

Re: Show HN: Minimal build system using just /bin/sh

#41

Post by chubot » Fri, Aug 18, 2017, 1:21 PM UTC

adtac wrote: post #13

vog wrote: post #11

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error h…

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.

Hash tables are useful for expressing build variants -- some examples at the end of this comment:

https://news.ycombinator.com/item?id=15043633

debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.

Re: Show HN: Minimal build system using just /bin/sh

#42

Post by brudgers » Fri, Aug 18, 2017, 1:21 PM UTC

edejong wrote: post #33

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA). No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG…

There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository.

Alternatives are not replacements.

Re: Show HN: Minimal build system using just /bin/sh

#43

Post by edejong » Fri, Aug 18, 2017, 1:30 PM UTC

antibjarne wrote: post #40

edejong wrote: post #33

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA). No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG…

Maybe they're not using a language with insanely bloated compile times?

It's not just about compile time. Couple of examples:

- Parallel downloading / uploading of artifacts

- Multi-step transformation (old-style: lexer to C to Object file to linker to deployment). You can turn this around, but generally you want each output of your lexer to be input to the C compiler and so on.

- Testing (especially brute-force testing might take time)

And in modern situations you have the same, with Typescript => Javascript => Minimisation => Combination => Deployment (roughly).

It really helps to think of this as a dependency chain and use a partial-order planner to find a possible linearization and execute.

Re: Show HN: Minimal build system using just /bin/sh

#44

Post by edejong » Fri, Aug 18, 2017, 1:33 PM UTC

brudgers wrote: post #42

edejong wrote: post #33

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA). No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG…

There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository. Alternatives ar…

Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones.

A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, CMake, Maven, SBT and many others.

Re: Show HN: Minimal build system using just /bin/sh

#45

Post by majewsky » Fri, Aug 18, 2017, 2:12 PM UTC

vog wrote: post #11

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error h…

I see you've not yet encountered Alpine-based Docker containers.

  $ docker run --rm -ti alpine sh             
  / # bash
  sh: bash: not found
  / # python
  sh: python: not found
  / # perl
  sh: perl: not found
  / # ruby
  sh: ruby: not found

Sure, you can install all of these with a single `apk add`, but it makes your container much fatter, i.e. every CI job that has to pull the image will be taking a longer time. Same for when you push the image into a far-away datacenter.

Re: Show HN: Minimal build system using just /bin/sh

#46

Post by moosingin3space » Fri, Aug 18, 2017, 2:28 PM UTC

edejong wrote: post #44

brudgers wrote: post #42

Earlier quoted context omitted.

There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository. Alternatives ar…

Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones. A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, C…

To add to this, Google spent a lot of time and effort on building a custom build system (Bazel) to make this sort of dependency tracking in a mono-repo easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell scripts any day, but the OP appears to have been focusing on extremely simple projects, where shell scripts are okay.

Re: Show HN: Minimal build system using just /bin/sh

#47

Post by swiley » Fri, Aug 18, 2017, 2:35 PM UTC

vog wrote: post #11

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error h…

Python never works right for me, having to deal with yet another dependency just to use the build system is obnoxious.

Just use make. It's not hard and it works even for fairly large projects.

Re: Show HN: Minimal build system using just /bin/sh

#48

Post by brudgers » Fri, Aug 18, 2017, 2:36 PM UTC

edejong wrote: post #44

brudgers wrote: post #42

Earlier quoted context omitted.

There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository. Alternatives ar…

Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones. A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, C…

My apologies for not expressing myself as clearly as I would like.

Some systems don't need to scale. Make and SBT etc. have an impedance mismatch with some projects due to their learning curve and the current sophistication of the development team.

The most important feature of any development tool is not whether it is best practice, it is whether it works. For example, Tarn Adams doesn't use version control for Dwarf Fortress. [1] That doesn't mean Microsoft should have switched the Windows codebase to zip files instead of Git any more than a single developer should start boiling the small lake of SBT just because it is better.

Sometimes, it is easier and faster and better to just build the tool that is needed. And of course sometimes it isn't.

[1]: https://www.reddit.com/r/IAmA/comments/1avszc/im_tarn_adams_...

Re: Show HN: Minimal build system using just /bin/sh

#49

Post by kingmanaz » Fri, Aug 18, 2017, 2:36 PM UTC

nailer wrote: post #34

vog wrote: post #11

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error h…

Also most current Unixes have at least Bash. Bourne shell is surprisingly rare outside BSD.

Or Korn, which is better than Bash, IMHO.

Re: Show HN: Minimal build system using just /bin/sh

#50

Post by brudgers » Fri, Aug 18, 2017, 2:37 PM UTC

moosingin3space wrote: post #46

edejong wrote: post #44

Earlier quoted context omitted.

Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones. A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, C…

To add to this, Google spent a lot of time and effort on building a custom build system (Bazel) to make this sort of dependency tracking in a mono-repo easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell scripts any day, but the OP appears to have been focusing on extremely simple projects, where shell scripts are okay.

Right, "works for me" doesn't mean "you must do it.".