Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JJ: JSON Stream Editor

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: JJ: JSON Stream Editor

#41

Post by Willuminaughty »

Hey there,

Just wanted to drop a quick note to say how much I'm loving jj. This tool is seriously a game-changer for dealing with JSON from the command line. It's super easy to use and the syntax is a no-brainer.

The fact that jj is a single binary with no dependencies is just the cherry on top. It's so handy to be able to take it with me wherever I go and plug it into whatever I'm working on.

And props to you for the docs - they're really well put together and made it a breeze to get up and running.

Keep up the awesome work! Can't wait to see where you take jj next.

Cheers

Re: JJ: JSON Stream Editor

#42

Post by notorandit »

untech wrote: post #37
notorandit wrote: post #33

Interesting. How often do you manipulate a 1+MB JSON file? Maybe I am wrong, but going from 0.01s to 0.001s doesn't motivate me to switch to jj.

Datasets are often stored in (sometimes gzipped) jsonlines format in my field (NLP). The file size could reach 100s of GBs.

100s of GBs?

In those cases, querying un-indexed files seems quite a thinko. Even if you can fit it all in RAM.

If you only scan that monstrous file sequentially, then you don't need either jq or jj or any other "powerful" tool. Just read/write it sequentially.

If you need to make complex scans and queries, I suspect a database is better suited.

Re: JJ: JSON Stream Editor

#43

Post by mi_lk »

j1elo wrote: post #23

I'll take the chance to bring attention to the maintenance issues that 'jq' has been having in the last years [1]; there hasn't been a new release since 2018, which IMO wouldn't necessarily be a bad thing if not for the fact that the main branch has been collecting improvements and bug fixes [2] since then. A group of motivated users are currently talking about what direction to take; a fork is being considered in or…

Looks like it's because @stedolan goes silent and not delegating the right GitHub repo accesses to the existing maintainers.

He seems to be working at Jane Street though, so if anyone is able to reach him please help the jq community :)

https://signals-threads.simplecast.com/episodes/memory-manag...

Re: JJ: JSON Stream Editor

#44

Post by notorandit »

capableweb wrote: post #25
vmfunction wrote: post #11

Earlier quoted context omitted.

that is what gets me, why did the file get to 20g? At that point just ship a SQLite file.

Does it matter why? Sometimes files gets big, and you don't control the generation or trying to change the generation is a bigger task than just dealing with a "big" (I'd argue 20GB isn't that big anyways) file with standard tools.

Nope, it matters a lot! Unstructured unindexed files get that gig usually as the result of some design flaw.

Re: JJ: JSON Stream Editor

#45

Post by orev »

Calzifer wrote: post #31
capableweb wrote: post #24

Earlier quoted context omitted.

What exactly is missing/broken in jq right now which warrants a fork? I've been using jq daily for years, and I can't remember the last time I hit a bug (must have been many years ago) and I can't recall any features I felt been missing for the years I've been using it. For me it's kind of done. It could be faster, but then I tend to program a solution myself instead, otherwise I feel like it's Done Enough.

What I miss from jq and what is implemented but unreleased is platform independent line delimiters. jq on Windows produces \r\n terminated lines which can be annoying when used with Cygwin / MSYS2 / WSL. The '--binary' option to not convert line delimiters is one of those pending improvements. https://github.com/stedolan/jq/commit/0dab2b18d73e561f511801...

You’ll have a much better experience in Cygwin/MSYS2/WSL if you treat them like isolated environments and not call programs from outside of them. If you want to use ‘jq’ (or any tool) within Cygwin, install the Cygwin package. Don’t rely on the Windows install, and you’re guaranteed to run into problems like this.

Re: JJ: JSON Stream Editor

#46

Post by dicknuckle »

bobnamob wrote: post #29
lathiat wrote: post #7

Earlier quoted context omitted.

The limiting speed factor of jq for me is, by far, figuring out how to write the expression I need to parse a fairly small amount of data. I do a bunch of support analysis and often writing a one-liner to put into a shell script to extract some bit of JSON to re-use later in the script. Often this is going to be used only once by me or a customer to run some task. Followed closely by figuring out the path to the area…

I’m far more likely to parse json into clojure repl session and go from there these days. Learning jq for the odd json manipulation I need to do seems like overkill

For me it's usually for some automation task to gather a list of IDs for some cloud environment to build infra things.

Re: JJ: JSON Stream Editor

#47

Post by nigeltao »

zokier wrote: post #30
nigeltao wrote: post #8

Earlier quoted context omitted.

jj is faster than jq. However, jsonptr is even faster and also runs in a self-imposed SECCOMP_MODE_STRICT sandbox (very secure; also implies no dynamically allocated memory). $ time cat citylots.json | jq -cM .features[10000].properties.LOT_NUM "091" real 0m4.844s $ time cat citylots.json | jj -r features.10000.properties.LOT_NUM "091" real 0m0.210s $ time cat citylots.json | jsonptr -q=/features/10000/properties/LOT…

Looks neat. One suggestion: add better build instructions on wuffs readme/getting started guide. I jumped in and tried to build it using the "build-all.sh" script that seemed convenient, but gave up (for now) after nth build failure due yet another missing dependency. It's extra painful because the build-all.sh is slow, so maybe also consider some proper build automation tool (seeing this is goog project, maybe bazel…

Thanks for the feedback. I'll add better build instructions.

If you just want the jsonptr program, instead of everything in the repo (the Wuffs compiler (written in Go), the Wuffs standard library (written in Wuffs), tests and benchmarks (written in C/C++), etc) then you can use "build-example.sh" instead of "build-all.sh".

  ./build-example.sh example/jsonptr
For example/jsonptr, that should work "out of the box", with no dependencies required (other than a C++ compiler). For e.g. example/sdl-imageviewer, you'll also need the SDL library.

Alternatively, you could just invoke g++ directly, as described at the very top of the "More details are at [link]" page in the grand-parent comment.

  $ git clone https://github.com/google/wuffs.git
  $ g++ -O3 -Wall wuffs/example/jsonptr/jsonptr.cc -o my-jsonptr

Re: JJ: JSON Stream Editor

#48

Post by zimpenfish »

maleldil wrote: post #21

Am I correct in understanding that this can only manipulate (get or set values) from a JSON path? That is, is it not a replacement for jq? For example, I frequently use jq for queries like this: jq '.data | map(select(.age Or this: jq '.data | map(.country) | sort[]' input.json | uniq -c Is it possible to do something similar with this tool? This is not a slight at jj. Even if it's more limited than jq, it's still of…

Annoyingly, I think `jq` might still be the only tool capable of these kinds of things. The rest seem to be "query simple paths and print the result" (which is handy, of course - I often use `gron` to get an idea of the keys I'm after because the linear format is easier to handle than JSON.)

Re: JJ: JSON Stream Editor

#49

Post by untech »

notorandit wrote: post #42
untech wrote: post #37

Earlier quoted context omitted.

Datasets are often stored in (sometimes gzipped) jsonlines format in my field (NLP). The file size could reach 100s of GBs.

100s of GBs? In those cases, querying un-indexed files seems quite a thinko. Even if you can fit it all in RAM. If you only scan that monstrous file sequentially, then you don't need either jq or jj or any other "powerful" tool. Just read/write it sequentially. If you need to make complex scans and queries, I suspect a database is better suited.

Usually you indeed scan this file sequentially, doing some filtration / transformation. As you do this transformation for each record, the speed of the tool used (e.g. jq) really matters.

Databases are not used in this case because it’s a complexity overhead compared to plain-text files. The ability to use unix pipelines and tools (such as grep) is a bonus.

Re: JJ: JSON Stream Editor

#50

Post by TymekDev »

maleldil wrote: post #21

Am I correct in understanding that this can only manipulate (get or set values) from a JSON path? That is, is it not a replacement for jq? For example, I frequently use jq for queries like this: jq '.data | map(select(.age Or this: jq '.data | map(.country) | sort[]' input.json | uniq -c Is it possible to do something similar with this tool? This is not a slight at jj. Even if it's more limited than jq, it's still of…

It looks like the README in jj repository does not do justice when it comes to available syntax for queries. jj uses gjson (by the same author) and its syntax [0]. From what I saw the first one can be handled with:

    jj 'data.#(age
I don't think there is a way to sort an array, though. However, there is an option to have keys sorted. Personally, I don't think there is much annoyance in that. One could just pipe jj output to `sort | uniq -c`.

I just discovered that gjson supports custom modifiers [1]. So technically, you could fork jj, and add another file registering `@sort` modifier via `gjson.AddModifier` and have a custom jj version supporting sorting.

[0]: https://github.com/tidwall/gjson/blob/master/SYNTAX.md

[1]: https://github.com/tidwall/gjson/blob/master/SYNTAX.md#modif...

Post reply on HN