51–60 of 62 posts
Earlier quoted context omitted.
For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk. Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
Can you explain more or provide more information/links?
Contemplating this, it’s too bad the Unix scripting ecosystem never evolved a tripartite symbiosis of ‘file‘, ‘lex‘, and ‘yacc‘, or similar tools. That is, one tool to magically identify a file type, one to tokenize it based on that identification, one to correspondingly parse it. All in a streaming/pipe-friendly mode. Would fit right in, other than the Unix prejudice (nonsensical from Day 0) for LF-separated text re…
I guess ... I don't think it really gets you much unless it's 1-pass streaming otherwise we're dealing with entire input buffers and then we're just back to files. You could argue using the vertical separator | is more syntactically graceful but then it's just a shell argument. There's quite a few radically different shells out there these days like xonsh, murex, and nushell so if simply arranging logic on the screen…
JSON is not a friendly format to the Unix shell — it’s hierarchical, and cannot be reasonably split on any character Yes, shell is definitely too weak to parse JSON! (One reason I started https://oils.pub is because I saw that bash completion scripts try to parse bash in bash, which is an even worse idea than trying to parse JSON in bash) I'd argue that Awk is ALSO too weak to parse JSON The following code assumes th…
> Yes, shell is definitely too weak to parse JSON! Parsing is a trivial, rejecting invalid input is trivial, the problem is representing the parsed content in a meaningful way. > bash completion scripts try to parse bash in bash You're talking about ble.sh, right? I investigated it as well. I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing w…
No, the complexity of the parser can be attributed to the incremental parsing. ble.sh implements an incremental parser where one can update only the necessary parts of the previous syntax tree when a part of the command line is modified. I'd probably use the same data structure (but better abstracted using classes) even if I could implement the parser in C or in higher-level languages.
Earlier quoted context omitted.
> Yes, shell is definitely too weak to parse JSON! Parsing is a trivial, rejecting invalid input is trivial, the problem is representing the parsed content in a meaningful way. > bash completion scripts try to parse bash in bash You're talking about ble.sh, right? I investigated it as well. I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing w…
> I think they made some choices that eventually led to the parser being too complex, largely due to the problem of representing what was parsed. No, the complexity of the parser can be attributed to the incremental parsing. ble.sh implements an incremental parser where one can update only the necessary parts of the previous syntax tree when a part of the command line is modified. I'd probably use the same data struc…
I use flex. Faster than awk or python. It produces relatively seems to be available "everywhere" because it is a build requirement for so many software programs. For exampole, NetBSD toolchain includes it. It is a build requirement for Linux kernel.^1 I have even used it with Interix SFU on Windows before WSL existed. I do not use jq. Too complicated for me. Overkill. I created statically-linked program less than hal…
Earlier quoted context omitted.
You may well already be aware, but just in case you aren't, your bin-true benchmark mostly measures dynamic loader overhead, not fork-exec (e.g., I got 5.2X faster using a musl-gcc statically linked true vs. glibc dynamic coreutils). { Kind of a distro/cultural thing what you want to measure (static linking is common on Alpine Linux, BSDs, less so on most Linux), but good to know about the effect. }
Yup, I added an osh-static column there, because I know dynamic linking slows things down. (With the latest release, we have a documented build script to make osh-static, which I tested with GNU libc and musl: https://oils.pub/release/latest/doc/help-mirror.html ) Although I think the CALLING process (the shell) being dynamically linked affects the speed too, not just the CALLED process (/bin/true) I'd like to read a…
One could probably set up a simple linear regression to get a good estimate of added cost-per-loaded .so on various OS-CPU combos, but I am unaware of a write up of such. It'd be a good assignment for an OS class, though.
Earlier quoted context omitted.
I guess ... I don't think it really gets you much unless it's 1-pass streaming otherwise we're dealing with entire input buffers and then we're just back to files. You could argue using the vertical separator | is more syntactically graceful but then it's just a shell argument. There's quite a few radically different shells out there these days like xonsh, murex, and nushell so if simply arranging logic on the screen…
What I meant was something like the SAX streaming parse model disaggregated and broken into sniff / lex / parse phases all mediated by a modestly structured stream of the kind a tool like awk could process naturally.
It's really about multiplexed semantic routing.
There's no great tools, abstractions, jargon or syscalls for it.
I mean there's plenty for things like the networking stack, dbus, Wayland, and various audio stacks but they're all a pain to deal with.
Nobody has really figured it out. The list inputs, list outputs, create bridges and taps, assign properties, which is what everybody does is complicated.
I'm sure there's brilliant people that find this intuitive but for me it requires too much orchestration and feels brittle.
People see this in microservices as well.
There needs to be something as intuitive as the mouse drag (Raskin), Unix pipe (McIlroy) or drop down menu (Atkinson) to deal with this stuff.
All these "obvious" things had to be invented. There's something dumb and obvious here nobody's cracked yet
The primary problem is the concept of discovering such things has ossified and few people are experimenting on introducing new ones
One day I wanted to use a TAP parser for the Test Anything Protocol. But I didn't want to be bogged down by dependencies.. so I didn't want to go nowhere near Python (and pyenv.. and anaconda.. and then probably having to dockerize that for some reason too..) nor nodeJS nor any of that. Found a bash shell script to parse TAP written by ESR of all people. That sounds fine, I thought. Most everywhere has bash, and ther…
For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk. Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
I find that that is one more environment where awk scripting can get the job done, python/perl/php/etc just can't be introduced, bash can _sometimes_ get the job done if it doesn't have to spawn too many subprocesses, and C/other-compiled-options _might_ be able to help if I had some kind of build environment targeting the platform(s) in question and enough patience.
I'll keep an eye out for python with no extra dependency options on the platforms that can handle that though.
Earlier quoted context omitted.
I was referring to the bash-completion project, the default on Debian/Ubuntu - https://github.com/scop/bash-completion/ But yes, ble.sh also has a shell parser in shell, although it uses a state machine style that's more principled than bash regex / sed crap. --- Also, distro build systems like Alpine Linux and others tend to parse shell in shell (or with sed). They often need package metadata without executing packa…
> uses a state machine style That's the way to go. I don't even consider other shallow and ad-hoc approaches as actually parsing it. I've been working on a state-machine based parser of my own. It's hard, I'm targetting very barebones interpreters such as posh and dash. Here's what it looks like https://gist.github.com/alganet/23df53c567b8a0bf959ecbc7b689... (not fully working example, but it gives an idea of what pu…
And I did notice that zsh can be incredibly slow -- just the parser itself is many times slower than other shells
A few years ago a zsh dev came on Zulip and wished us luck, probably because they know the zsh codebase has a bunch of technical debt
i.e. these codebases are so old that the maintainers only have so much knowledge/time to improve things ... Every once in awhile I think you have to start from scratch :)
Earlier quoted context omitted.
For the record, "python-without-extra-dependencies" is a thing and a very nice one too. I always prefer it over awk. Highly recommend to everyone - plenty of "batteries" included, like json parser, basic http client and even XML parser, and no venv/conda required. Very good forward compatibility. Fast (compared to bash).
That does sound compelling, but I frequently enough have to work on embedded systems (often running busybox) where python isn't installed and available R/W storage space is measured in tens or hundreds of Kb. I find that that is one more environment where awk scripting can get the job done, python/perl/php/etc just can't be introduced, bash can _sometimes_ get the job done if it doesn't have to spawn too many subproc…