Earlier quoted context omitted.
I've been using DJB's stuff for ~20 years and I don't like his recent build systems either. Without defending it, I'd just like to offer a theory on why he packages things the way he does. Back in the day, his build processes were atypical but still much more "normal" than now. He also released his software without licenses. During this time of heavy software development, DJB was concerned about people screwing aroun…
Appreciate the response! Interesting to read. The Libsodium guys wound up doing exactly what you're suggesting, because of the impossibility of trying to package NaCl as-is. So they essentially had to re-do/duplicate all of his build work just to make it packageable. And now there are two competing implementations (three if you count tweetnacl). And a bit of a confusing mess in the documentation department. It seems…
Djbsort: A new software library for sorting arrays of integers
121–130 of 158 posts
Re: Djbsort: A new software library for sorting arrays of integers
#122Earlier quoted context omitted.
You example script with python2 is not about pathing, but a separate problem or multiple binaries with the same name, and is exactly why you want to let package managers deal with this. As a developer, you can't know all the different setups you users will have, or for example what the default python will be. As a package manager, you know Exactly this for the systems you are packaging for. If needed, they will patch…
> You example script with python2 is not about pathing, but a separate problem or multiple binaries with the same name, and is exactly why you want to let package managers deal with this. > for example what the default python will be Except that package maintainers created this problem. It's not a real problem! Python's source code downloaded packages call itself python3. Some package managers decided to call it "pyt…
Re: Djbsort: A new software library for sorting arrays of integers
#123The median sort time for sorting 1048576 elements wth djbsort is 61467822 cycles: https://sorting.cr.yp.to/speed.html On a say 3.6 Ghz processor that would be around 17ms. So the number of elements sorted per second would be around 61.4 M elements/s. My parallel radix sort can sort floats (a little harder than integers) at around 165 M elements/s: http://forwardscattering.org/post/34 A serial radix sort should still…
My single-threaded floating point sort is also twice as fast as djbsort: http://stereopsis.com/radix.html This one uses 11-bit radix to save memory bandwidth. Without all the floating-point stuff it would be faster.
Re: Djbsort: A new software library for sorting arrays of integers
#124Earlier quoted context omitted.
Three of the source code files contain a notice that they are in the public domain: * cpucycles/mips/cpucycles.c * cpucycles/cortex_vct/cpucycles.c * cpucycles/cortex/cpucycles.c Regarding the missing license: I guess if you download the software from his website you are not allowed to distribute it yourself. Is that correct?
> Three of the source code files contain a notice that they are in the public domain: Making it legally dodgy to dangerous in mainland europe, either way certainly not reliably licensed.
Re: Djbsort: A new software library for sorting arrays of integers
#125My job involves a lot of packaging/cross-compilation, and djb's libraries always seem consistently hostile to the lowly packaging engineer. Would it really be all that much work to package in autotools or CMake? Why do I need his special-snowflake build system with its hard-coded assumptions about system paths? I know that the cult of djb will downvote this into oblivion, but seriously, what is the rationale for a bu…
Do you know of any guidelines for making libraries easy to package? Or examples of projects that do it right?
CPPFLAGS = # e.g. -I and -D preprocess options
CFLAGS = # various ad hoc compiler switches
LDFLAGS = # e.g. -L options
LDLIBS = # e.g. -l options
SOFLAGS = # -shared or -bundle -undefined dynamic_lookup
(NOTE: SOFLAGS is less standard--i.e. not in GNU style guide or GNU make manual--but perfectly fits the pattern and extremely useful to specify independently. You'll often find a variable like this used, if not the same name. IME SOFLAGS it the most common name.)Library dependencies should follow a similar pattern:
LIBFOO_CPPFLAGS = # e.g. -I/usr/local/foo/include
LIBFOO_CFLAGS =
LIBFOO_LDFLAGS = # e.g. -L/usr/local/foo/lib
LIBFOO_LDLIBS = -lfoo
To make it easier to control feature-specific options without overriding all other defaults, you should do likewise for feature groups: PIC_CFLAGS = -fPIC
...
PTHREAD_LDLIBS = -lpthread
...
WARN_CFLAGS = -Wall -Wextra
If you want to get really fancy, you can use top-level MY* variables so people can easily append flags without overriding existing variables.Then for most basic target recipes you can pre-compose all the flags to make your rules simpler:
ALL_CPPFLAGS = $(PIC_CPPFLAGS) $(LIBFOO_CPPFLAGS) $(WARN_CPPFLAGS) $(CPPFLAGS) $(MYCPPFLAGS)
ALL_CFLAGS = $(PIC_CFLAGS) $(LIBFOO_CFLAGS) $(WARN_CFLAGS) $(CFLAGS) $(MYCFLAGS)
ALL_LDFLAGS = $(PIC_LDFLAGS) $(LIBFOO_LDFLAGS) $(WARN_LDFLAGS) $(LDFLAGS) $(MYLDFLAGS)
ALL_LDLIBS = $(MYLDLIBS) $(LDLIBS) $(LIBFOO_LDLIBS) $(WARN_LDLIBS) $(PTHREAD_LDLIBS)
.o.c:
$(CC) $(ALL_CFLAGS) $(ALL_CPPFLAGS) -c -o $@ $
Because of the ubiquity of GCC and clang, and because of the commonality across most modern Unix platforms (e.g. SOFLAGS will be "-shared" on all BSD- and SysV-derived systems, including Linux, and "-bundle -undefined dynamic_lookup" on macOS), the above will allow people to build your software without modifying the actual Makefile in almost all scenarios. It will certainly make packagers' jobs easier.For installation you should always use the standard prefixes
DESTDIR = # temporary root to assist in packaging
prefix = /usr/local
bindir = $(prefix)/bin
datadir = $(prefix)/share
includedir = $(prefix)/include
libdir = $(prefix)/lib
sysconfdir = $(prefix)/etc
and install targets should be written like $(DESTDIR)$(libdir)/libacme.so: libacme.so
...
This may seem like alot of boilerplate, but the important thing is that it's very simple and time-tested. Autogenerating just this basic boilerplate is overkill and adds unnecessary indirection and complexity. You can get fancier than this scheme, but avoid the temptation as much as you can unless it's an obvious win--remedies a regular, concrete issue that is actually resulting in significant wasted effort. For complex builds (e.g. multiple targets where flags might not have the same semantics across target rules), consider splitting the build into multiple Makefiles. You can create a convenience wrapper, but at least the sub Makefiles will be reusable by packagers or contributors without having to modify anything directly. It's not ideal but there is no ideal solution. (I once heavily advocated non-recursive make, but these days I'm much more averse to complexity.)There's no easy answer for Windows, particularly because NMake uses significantly different syntax for some constructs. For really basic builds the above might actually work as long as you stick to NMake-compatible syntax. With the advent of Windows Subsystem for Linux (WSL) you could consider requiring WSL for the build (so you have access to POSIX-compatible shell and Make) and perhaps providing a wrapper Makefile with more convenient defaults for compiling with Visual Studio (which is very much still a command-line driven compiler). Requiring clang-cl might be even easier.
For simple stuff IME autotools and CMake are just overkill.[1] For complex stuff they often won't be sufficient, anyhow.[2] So the important thing is to layer your build so the maximum amount of build code can be reused by somebody with unforeseen needs; and reused in a way that is obvious and transparent so their tweaks can be simple. Stick to convention as much as possible. Even if you don't use autotools, autotools has created de facto conventions wrt build variables that will be intuitive to all packagers. Please don't invent your own naming scheme! (It's annoying when someone uses 'PREFIX' instead of 'prefix'. There's logic to the autotools scheme. Uppercase variable are for [build] flags used in rule recipes, lowercase for [installation] paths used to define rule targets and dependencies. Exceptions like DESTDIR are well-known and few.)
Always be mindful of the needs of people doing embedded work--cross-compiling, complex build flags, etc. But don't bend over backwards. All you really need to do is remove obstructions and the necessity to patch. They're used to writing build wrappers, but patching is a maintenance nightmare and impedes tracking upstream.
Ideally you would support out-of-tree builds, but in reality few packages support this properly (even autotools or CMake projects), and if you're not regularly doing out-of-tree builds your build probably won't work out-of-the-box anyhow. That's why I didn't mention the use of macros like VPATH, top_srcdir, etc, in your rules. Do that at your discretion, but beware of descending down the rabbit hole.
The same thing applies to automatic feature detection--unless you're regularly building on esoteric targets your feature detection is unlikely to work out of the box, and bit rot happens quickly as platforms evolve.[3] Don't put too much effort into this; just make sure it's easy to manually toggle feature availability and that the defaults work on the most common platforms and environments. Maximize benefits, minimize costs, but don't think you can automate everything away.
All those fancy builds people tout are ridiculously brittle in reality. People just don't realize it because they're only using the builds on what are in fact relatively homogenous environments. Most of the effort is wasted, and build complexity (including bespoke build environments) is a big reason why people don't reuse software as much as they could.
[1] Indeed, for simple builds it's not much of a burden to provide and maintain a POSIX build and a Visual Studio project file build.
[2] For CMake this is especially true. For feature detection CMake often requires actually updating your version of CMake. That's a gigantic PITA. Anyhow, most feature detection can be done inline using the preprocessor, or by simply defaulting to being available (this is 2018, not 1988--there's much more commonality today.) And as long as your feature-detection macros are controllable from build-time flags, people can easily work around issues by explicitly toggling availability. Thus, the autotools convention of using "#ifdef FEATURE_FOO" for feature flags is bad advice. Always use boolean flags like "#if FEATURE_FOO", and only define FEATURE_FOO in config.h-like files if not already defined. That allows people to do, e.g., "MYCPPFLAGS=-DFEATURE_FOO=0" to override feature detection as necessary.
[3] Autoconf's feature checking (check the feature, not the version) sounds robust in practice, but you'd be surprised how many ways it can break, or how difficult writing a properly robust feature check is. Plus, especially on Linux people expect a feature to work at run-time if the kernel supports it, regardless of whether glibc had a wrapper at build time or has it at run time. These issues are where you'll spend much of your time, and this can't be automated. You can't foresee all these issues! Focus on making it easy to supplement or work around your build!
Re: Djbsort: A new software library for sorting arrays of integers
#126Earlier quoted context omitted.
> Three of the source code files contain a notice that they are in the public domain: Making it legally dodgy to dangerous in mainland europe, either way certainly not reliably licensed.
I hear people raise this concern a lot, but I think it is without foundation. If something is in the public domain in the U.S. then anyone can use it for any purpose, including releasing it under whatever license they want. Of course, any constraints imposed by that license will be unenforceable since any user of the software can claim to be using it under the terms of some other license or, of course, as part of the…
However if me and my company and everything is in Europe and I use/redistribute the code in Europe I must follow applicable European law. If that doesn't accept that form of public domain the author (rights owner) could sue me and it were upon the judge, who sensible they are. (This is mostly theoretical - if the author decides to put it in public domain per U.S. law they most likely don't want to restrict to U.S.)
For an example see the recent case about project Gutenberg https://news.ycombinator.com/item?id=16511038
Re: Djbsort: A new software library for sorting arrays of integers
#127Earlier quoted context omitted.
I hear people raise this concern a lot, but I think it is without foundation. If something is in the public domain in the U.S. then anyone can use it for any purpose, including releasing it under whatever license they want. Of course, any constraints imposed by that license will be unenforceable since any user of the software can claim to be using it under the terms of some other license or, of course, as part of the…
Anybody under U.S. jurisdiction can follow U.S. law. However if me and my company and everything is in Europe and I use/redistribute the code in Europe I must follow applicable European law. If that doesn't accept that form of public domain the author (rights owner) could sue me and it were upon the judge, who sensible they are. (This is mostly theoretical - if the author decides to put it in public domain per U.S. l…
But if this really concerns you, I would be happy to provide you -- or anyone else -- with a licensed copy of any of DJB's code for a modest processing fee.
Re: Djbsort: A new software library for sorting arrays of integers
#128Re: Djbsort: A new software library for sorting arrays of integers
#129Earlier quoted context omitted.
I actually really like the authenticity and humility of DJB including that in the instructions. I think it's likely many people trust his code (and he's certainly written a lot of extremely security sensitive stuff), but of course it's a much better practice to not trust him quite so much.
Authenticity, sure. Humility - not after his approach to the students issue in recent years where he was more interested in being correct then helping people :-(
That does describe him pretty well.
Re: Djbsort: A new software library for sorting arrays of integers
#130Earlier quoted context omitted.
For what it's worth, I agree that Autoconf tests for a lot of things that aren't necessarily relevant today. I don't think I need to worry about the existence of on just about any platform. But there ARE a lot of very relevant tests mixed in there too. These include things like: - Width of an int - Endianness - Availability of arbitrary functions for linking - Search through a list of libraries until one is found tha…
> - Width of an int If your software depends on this, you're doing it wrong. As you said, depending on the existence of is just fine, and you can then specify what you need. Even in the incredibly rare case of needing the width of int (serializing inputs and outputs of existing libraries interoperably on multiple machine ABIs) has you covered, albeit in an awkward way, unless you can assume POSIX and thus have WORD_B…