Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB as the New jq

pgrs.net

Post reply on HN
Print view View original

21–30 of 74 posts

Re: DuckDB as the New jq

#21

Post by xg15 »

pcthrowaway wrote: post #20
xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places After a few years of stalled development, jq has been taken over recently by a new team of maintainers and is rapidly working through a lot of longstanding issues ( https://github.com/jqlang/jq ), so I'm not sure if this is still the ca…

Wasn't aware of that, that's great to hear! I think if there is one utility that deserves a great maintainer team then this one. But if we saw some actual improvements in the future, that would be awesome!

I have a list of pet peeves that I'd really like to see fixed, so I'm gonna risk a bit of hope.

Re: DuckDB as the New jq

#22

Post by esafak »

NortySpock wrote: post #4

In a similar vein, I have found Benthos to be an incredible swiss-army-knife for transforming data and shoving it either into (or out of) a message bus, webhook, or a database. https://www.benthos.dev/

I wish it was not based on YAML. Pipelines are code, not configuration!!

Re: DuckDB as the New jq

#23

Post by jimbokun »

xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

The Unix philosophy continues to pass the test of time.

Re: DuckDB as the New jq

#24

Post by edu_guitar »

JeremyNT wrote: post #19

I have a lot of trouble understanding the benefits of this versus just working with json with a programming language. It seems like you're adding another layer of abstraction versus just dealing with a normal hashmap-like data structure in your language of choice. If you want to work with it interactively, you could use a notebook or REPL.

if you are used to the command line and knows some basic syntax, it is less verbose then opening a REPL and reading a file. The fact that you can pipe the json data into it is also a plus, making it easier to check quickly if the response of a curl call has the fields/values you were expecting. Of course, if you are more comfortable doing that from the REPL, you get less value from learning jq. If you are fond of one liners, jq offers a lot of potential.

Re: DuckDB as the New jq

#25

Post by ndr »

If you like lisp, and especially clojure, check out babashka[0]. This my first attempt but I bet you can do something nicer even if you keep forcing yourself to stay into a single pipe command.

  cat repos.json | bb -e ' (->> (-> *in* slurp (json/parse-string true))
                                (group-by #(-> % :license :key))
                                (map #(-> {:license (key %)
                                           :count (-> % val count)}))
                                json/generate-string
                                println)'
[0] https://babashka.org/

Re: DuckDB as the New jq

#26

Post by bdcravens »

JeremyNT wrote: post #19

I have a lot of trouble understanding the benefits of this versus just working with json with a programming language. It seems like you're adding another layer of abstraction versus just dealing with a normal hashmap-like data structure in your language of choice. If you want to work with it interactively, you could use a notebook or REPL.

Pipelining CLI commands or bash scripts. From a security perspective, it may be preferable to not ship with a runtime.

Re: DuckDB as the New jq

#27

Post by eru »

xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it [...]

It's basically just functional programming. (Or what you would get from a functional programmer given the task of writing such a tool as jq.)

That's not to diminish jq, it's a great tool. I love it!

Re: DuckDB as the New jq

#28

Post by salmo »

xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

As an old Unix guy this is exactly how I see jq: a gateway to a fantastic library of text processing tools. I see a lot of complicated things done inside the language, which is a valid approach. But I don’t need it to be a programming language itself, just a transform to meet my next command after the pipe.

If I want logic beyond that, then I skip the shell and write “real” software.

I personally find those both to be more readable and easier to fit in my head than long complex jq expressions. But that’s completely subjective and others may find the jq expression language easier to read than shell or (choose your programming language).

Re: DuckDB as the New jq

#29

Post by vips7L »

bdcravens wrote: post #26
JeremyNT wrote: post #19

I have a lot of trouble understanding the benefits of this versus just working with json with a programming language. It seems like you're adding another layer of abstraction versus just dealing with a normal hashmap-like data structure in your language of choice. If you want to work with it interactively, you could use a notebook or REPL.

Pipelining CLI commands or bash scripts. From a security perspective, it may be preferable to not ship with a runtime.

bash and jq are both runtimes.

Re: DuckDB as the New jq

#30

Post by bdcravens »

vips7L wrote: post #29
bdcravens wrote: post #26

Earlier quoted context omitted.

Pipelining CLI commands or bash scripts. From a security perspective, it may be preferable to not ship with a runtime.

bash and jq are both runtimes.

Very difficult (or often impractical) to not have a shell at all, and jq is at least limited in scope, and has no dependencies that need to be installed. Far better than a full language with its own standard library and set of dependencies to lock down.
Post reply on HN