Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

github.com

Post reply on HN
Print view View original

1–10 of 56 posts

Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#1

Post by h4l »

jb is a UNIX tool that creates JSON, for shell scripts or interactive use. Its "one thing" is to get shell-native data (environment variables, files, program output) to somewhere else, using JSON encapsulate it robustly.

I wrote this because I wanted a robust and ergonomic way to create ad-hoc JSON data from the command line and scripts. I wanted errors to not pass silently, not coerce data types, not put secrets into argv. I wanted to leverage shell features/patterns like process substitution, environment variables, reading/streaming from files and null-terminated data.

If you know of the jo program, jb is similar, but type-safe by default and more flexible. jo coerces types, using flags like -n to coerce to a specific type (number for -n), without failing if the input is invalid. jb encodes values as strings by default, requiring type annotations to parse & encode values as a specific type (failing if the value is invalid).

If you know jq, jb is complementary in that jq is great at transforming data already in JSON format, but it's fiddly to get non-JSON data into jq. In contrast, jb is good at getting unstructured data from arguments, environment variables and files into JSON (so that jq could use it), but jb cannot do any transformation of data, only parsing & encoding into JSON types.

I feel rather guilty about having written this in bash. It's something of a boiled frog story. I started out just wanting to encode JSON strings from a shell script, without dependencies, with the intention of piping them into jq. After a few trials I was able to encode JSON strings in bash with surprising performance, using array operations to encode multiple strings at once. It grew from there into a complete tool. I'd certainly not choose bash if I was starting from scratch now...

Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON
github.com

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#2

Post by h4l »

As well as anyone's general thoughts/experiences, I'd appreciate opinions on the error handling mechanism jb uses to detect errors in upstream jb processes that jb is reading from.

Normally, detecting errors on the other end of a pipe requires care in a shell environment (e.g. retrospectively checking PIPESTATUS). I used an approach I've called Stream Poisoning. It takes advantage of the fact that control characters are never present in valid JSON. When jb fails to encode JSON, it emits a Cancel control character[1] on stdout. When jb encounters such a character in an input, it can tell the input it's reading from is truncated/erroneous. This avoids the typical problem of a pipe silently being read as an empty file.

I've got a page explaining this with some examples here: https://github.com/h4l/json.bash/blob/main/docs/stream-poiso... I can imagine using control characters in a text stream being rather controversial, but I feel it works quite well in practice.

[1]: https://en.wikipedia.org/wiki/Cancel_character

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#5

Post by pmarreck »

BATS was a little heavy for me as a testing dependency for my own use (I ended up writing what I intended to be "the most minimalist shell testing library possible", see below, I think it still needs work though!), but I at least want to commend you for having what looks like a great test suite to begin with!

https://github.com/pmarreck/tinytestlib

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#6

Post by mg »

I like the syntax to send typed values from the terminal:

    jb id=42 size:number=42 surname=null data:null

    => {"id":"42","size":42,"surname":"null","data":null}
I never had the need to use typed arguments in bash, but if I ever have it, this might be the syntax I'd use.

In fact, I was thinking about such a syntax recently. I am writing a tool which lets you call functions in Python modules from the command line. At first, I thought I need to define the argument types on the command line. But then I decided it is more convenient to use inspection and auto-convert the values to the needed types.

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#7

Post by altruios »

Windows: what if everything was an (command) object?

Linux: what if everything was a file?

Soon we might have...

Mong/Os: what if everything was JSON?

YiAM/OS: YiAM/OS is ANOTHER MARKUP OPERATING SYSTEM... would come out shortly thereafter...

I like JSON and getting in the terminal is a challenge - GOOD JOB!

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#8

Post by [deleted] »

altruios wrote: post #7

Windows: what if everything was an (command) object? Linux: what if everything was a file? Soon we might have... Mong/Os: what if everything was JSON? YiAM/OS: YiAM/OS is ANOTHER MARKUP OPERATING SYSTEM... would come out shortly thereafter... I like JSON and getting in the terminal is a challenge - GOOD JOB!

[deleted]

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#9

Post by enriquto »

mg wrote: post #6

I like the syntax to send typed values from the terminal: jb id=42 size:number=42 surname=null data:null => {"id":"42","size":42,"surname":"null","data":null} I never had the need to use typed arguments in bash, but if I ever have it, this might be the syntax I'd use. In fact, I was thinking about such a syntax recently. I am writing a tool which lets you call functions in Python modules from the command line. At fir…

> I like the syntax to send typed values from the terminal:

Incidentally, this syntax shows a notation that is clearly superior to json (at least for non-nested stuff). If all you need is this, you'd be better off by avoiding json altogether.

[Rant: if json is so unergonomic that people keep inventing alternatives like this syntax and stuff like "gron" to de-jsonise their lives, maybe using json was always a bad idea, after all... I guess in a decade everybody will look at json with the same disdain as we do XML today.]

Re: Show HN: Jb / json.bash – Command-line tool (and bash library) that creates JSON

#10

Post by h4l »

pmarreck wrote: post #5

BATS was a little heavy for me as a testing dependency for my own use (I ended up writing what I intended to be "the most minimalist shell testing library possible", see below, I think it still needs work though!), but I at least want to commend you for having what looks like a great test suite to begin with! https://github.com/pmarreck/tinytestlib

It's nice to have compact single-file dependencies like this! I like the look of your assertions (checking out, err & status). I definitely found myself writing my own assertions to get understandable errors.
Post reply on HN