>
100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters.In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters.
------
> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.
Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents:
command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»).
How the length limit works is explained at length (please pardon the pun) here:
https://www.in-ulm.de/~mascheck/various/argmax/------
> Excellent path parsing. […] finding the file extension […]
File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1). Moreover, any valid character can be used as a separator and its interpretation is either left out (almost always), or the interpretation is left up to the semantically aware app. One is free to use a comma or the Javanese wasana pada as part of the extension; that is, «file,exec» and «file꧅ ꦆ ꧅exec» are both valid and both have the extension of «exec» – as long as the file system supports the appropriate character set. This is also why «basename», to work correctly, requires the separator as part of the imaginary «extension», i.e. «basename myfile,exec ,exec» will always give «myfile» as the result.
DOS style extensions are a made up convention that neither the kernel, nor the shell, nor file processing utilities enforce as they are file extension unaware. It is better to think of UNIX file names as being made up a prefix and an optional, arbitrary length suffix (but not an extension).
------
> Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available
With respect to «every core available». You almost certainly don't want that and should overprovision the number of running processes compared to the number of cores on your system.
UNIX has supported multiprocessor systems for a very long time. Despite massive improvements in the hardware performance, disks and networks are still the slowest moving (or still) parts. They were even slower when UNIX was in its relative infancy when the CPU time was also very expensive. Therefore, the CPU time had to utilised efficiently whilst waiting for a disk to return a string of bytes.
File processing tasks spend their time between: 1) waiting for I/O (in the blocked state) and 2) actually processing (the running state). Since the disk is still very slow compared to the speed of a modern CPU, the UNIX process scheduler blocks the process until I/O completes and checks the process run queue in the kernel to see if there is another process ready to compute something (i.e. in the running state). This inherent interleaving of «blocked for I/O» and «running» process states can be used to an advantage depending on a few factors.
The less is size of the unit of data an app processes and the larger is the total size of the input (i.e. the input and output files), the more time the app spends in the «blocked for I/O» state, and most of the CPU time is simply wasted unless there is something else to do. But if we know such specifics (the size of the unit of work and the size of the input), we can overprovision the number of processes thereby utilising the CPU compute time more efficiently whilst the disk controller is transferring bytes into the memory via the direct memory access (I am oversimplifying a few bits here). This is the reason why «make -j12» will compile almost always faster than «make -j8» on a 8 CPU core system on projects with a large number of small(-er) files – because of the I/O overhead. Whilst there is no universal formula, 1.5x process overprovisioning factor is a decent starting point. For simpler daily file processing tasks GNU parallel is good enough to spare oneself of headaches of such computations, though.
Therefore, the «We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available […]» does not make sense in the context of UNIX shell languages and the process scheduling in UNIX. In fact, you will underutilise your 128 CPU cores, sometimes pretty heavily, unless you correctly account for the I/O factor.
Only if the app/process is aware of how to efficiently parallelise its own workload (because it knows its workload better than anyone else), then and only then it does need to know how many cores there are available at its disposal. No scripting language / shell can solve this problem.
------
> Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere".
The excellent error reporting has existed in UNIX since day 1. It is called the process status code; anything that is not a zero status code indicates an error. The semantic interpretation of each speific numeric status code, however, is entirely decoupled from the thing that might fail and is documented in the man page. The status code of 3 in «mv» and in «git» will mean two completely different things, therefore the specific status code processing is localised to the process invocation point in shell scripts. Most of the time, though, I personally don't want my shell script to explode with an error message from a random failed command unless it is something of extreme importance to me; checking for the process exit code and acting upon it accordingly is sufficient and is good enough.
Whether such an approach is a good thing or not is a matter of a debate. Global lists of errors and/or error messages require an official register of both diligently maintained and updates of which to be centrally coordinated, which I don't think could work with the open source.
Other operating systems have attempted to mandate error codes with complex structures and a well defined (and sometimes written by a professional technical writer!) error message. Yet, they have had limited success. Yes, OS/400 running on an AS/400 could inspect a failed process' error code and automatically dispatch a message to an IBM service centre to order a spare part for a specific failing piece of hardware whose SKU would have been deduced from a specic part of the error code without requiring the human intervention, but that is somewhat of an extreme and extravagant example and is certainly not mainstream.*
EDIT: «make -j12» vs «make -j8» explanation.