Great! Now improve the syntax!
Q: A faster re-implementaiton of jq written in Reason Native/OCaml
21–30 of 196 posts
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#22This looks interesting, but could be confusing given the programming language of the same name ( https://code.kx.com/q/ )
Ah yes, the old ".j.k raze read0`" as a separate app
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#23Earlier quoted context omitted.
jq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file). The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.
Out of interest, what created a 90GB JSON file?
Not OP, but I routinely call a specific HTTP API for millions of entities or pull down entire Kafka topics - all in JSON format. For various reasons those are the canonical sources of data and/or the most performant, so I end up ripping through GBs and GBs of JSON when troubleshooting/reporting on things.
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#24Do we need to make jq faster ? Anyone has issues with current speed ? Is there any specific reason other than "because we can" ?
I can't answer for the OP, but "because we can" is a valid enough reason (pun unintended) for me.
IMO, an individual dev making a fast useful tool should always be welcomed as a feat of worthy hacking.
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#25Are we sure it should get a single-letter 'q' binary name though? Docs seem to point that it's short for 'query-json'? Why not call it 'query-json' and let the user decide that as a shell alias or whatever. Even the ubiquitous 'ls' and 'cd' are two characters.
Yeah. Can you imagine trying to do a web search for ‘q’?
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#26As an outsider I get very confused by the Reason / Reason Native / OCaml / Bucklescript / Rescript?! ecosystem. What does it mean for it to be written in Reason Native/OCaml?
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#27Umm... There's already a language called Q for array processing.
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#28Is jq slow? I have only worked with datasets up to 1mb but I’ve never had a performance issue that wasn’t attributed to my error.
Yes. I was looking to embed it in a tool, but decided against it after looking at its implementation. It parses the expression with a stack and executes it directly, and its JSON parsing is much the same. I doubt the parsing would be close to competitive with RapidJSON, let alone simd-json. The conditionals and pointer chasing of such an implementation are stumbling blocks to performance.
The C code is clean enough as C code goes, but fairly monolithic. And it’s C, so it’s not noticeably slow until you start processing GB. But it would probably take a rewrite to improve its performance significantly.
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#29Great! Now improve the syntax!
How, though? I agree that jq's syntax isn't exactly the most straightforward, and it gets raised as a point of criticism anytime jq is mentioned, but its scripting language seems like a pretty good compromise between compactness and rich features.
Replacing that with, say, traditional command line flags would make it a lot less useful for me, I'd probably have to build much longer pipe-chains to do things that are relatively simple and readable jq snippets (if one knows the syntax.)
Using an established scripting language in its place would make it pretty much just python -c/ruby -e or whatever with some pre-loaded functions, but what's the point? You can always just write a quick python/ruby/whatever script, jq to me is an alternative for cases where a script feels unnecessary. It would also mean everything gets more verbose, so less of my jq transformations can be inlined without loss of readability.
Aligning it to more established languages would probably cause confusion as well in those cases where it doesn't match the reference language 1:1. Looks like javascript, writes like javascript, but only for a tiny subset of the language, etc.
Doing this only for a few function names or syntax constructs still results in a pretty unique and unusual language that will require people to reference the docs a lot, just now lots of existing scripts break.
Re: Q: A faster re-implementaiton of jq written in Reason Native/OCaml
#30Do we need to make jq faster ? Anyone has issues with current speed ? Is there any specific reason other than "because we can" ?
If we were at a board meeting deciding how to spend my time, no. If it is done? Why not