Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB as the New jq

pgrs.net

Post reply on HN
Print view View original

51–60 of 74 posts

Re: DuckDB as the New jq

#51

Post by Sammi »

I work primarily in projects that use js and I mostly don't see the point in working with json in other tools than js.

I have tried jq a little bit, but learning jq is learning a new thing, which is healthy, but it also requires time and energy, which is not always available.

When I want to munge some json I use js... because that is what js in innately good at and it's what I already know. A little js script that does stdin/file read and then JSON.parse, and then map and filter some stuff, and at the end JSON.stringify to stdout/file does the job 100% of the time in my experience.

And I can use a debugger or put in console logs when I want to debug. I don't know how to debug jq or sql, so when I'm stuck I end up going for js which I can debug.

Are there js developers who reach for jq when you are already familiar with js? Is it because you are already strong in bash and terminal usage? I think I get why you would want to use sql if you are already experienced in sql. Sql is common and made for data munging. Jq however is a new dsl when I don't see the limitation of existing js or sql.

Re: DuckDB as the New jq

#52

Post by pcthrowaway »

salmo wrote: post #50
pcthrowaway wrote: post #49

Earlier quoted context omitted.

> The Unix Programming Environment How does this compare to The Art of Unix Programming, if you've read both?

I don’t find that book to be very useful at all. I’m kind of annoyed by the bait and switch of the title. It’s a play on Knuth’s classic but then turns into showing why Unix/Linux is better than Windows, etc. As a disclaimer: I really don’t respect ESR and his work, and admire Brian Kernighan immensely. Very odd to be in a situation where those names are put side by side. Just want to call out that I do have bias on…

I wasn't aware of the bait and switch at the time I read it, but I did really enjoy the history of how the Unix/Linux ethic came together and evolved over time. Had I heard of The Unix Programming Environment when I read it in 2014 I may have gone with that instead, as I was looking for something more along the lines of a technical handbook rather than a code of ethics.

Re: DuckDB as the New jq

#53

Post by wwader »

Sammi wrote: post #51

I work primarily in projects that use js and I mostly don't see the point in working with json in other tools than js. I have tried jq a little bit, but learning jq is learning a new thing, which is healthy, but it also requires time and energy, which is not always available. When I want to munge some json I use js... because that is what js in innately good at and it's what I already know. A little js script that do…

I do quite a lot of adhoc/exploratory programming to query and transform data then jq is very convenient as it works very well with "deep" data structures and the language itself it very composable.

To debug in jq you can use the debug function to prints to stderr, ex: "123 | debug | ..." or "{a:123, b:456} | debug({a}) | ... " only prints value of a "{a:123}"

Re: DuckDB as the New jq

#54

Post by DanielHB »

xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

found out recently that jq can url-encode values, is there anything it _can't_ do?

Re: DuckDB as the New jq

#55

Post by salmo »

pcthrowaway wrote: post #52
salmo wrote: post #50

Earlier quoted context omitted.

I don’t find that book to be very useful at all. I’m kind of annoyed by the bait and switch of the title. It’s a play on Knuth’s classic but then turns into showing why Unix/Linux is better than Windows, etc. As a disclaimer: I really don’t respect ESR and his work, and admire Brian Kernighan immensely. Very odd to be in a situation where those names are put side by side. Just want to call out that I do have bias on…

I wasn't aware of the bait and switch at the time I read it, but I did really enjoy the history of how the Unix/Linux ethic came together and evolved over time. Had I heard of The Unix Programming Environment when I read it in 2014 I may have gone with that instead, as I was looking for something more along the lines of a technical handbook rather than a code of ethics.

Yeah and ESR can be revisionist in his history, projecting intention on something organic. He alienated a lot of people over time with this… and other behavior.

The book I recommended is both a handbook and a “how to think.” It applies forward to things introduced well after the book. But it also helped me understand why the Byzantine behavior of a tty is what it is.

If you are interested in the history from a first person perspective, I do recommend Kernighan’s “Unix: A History and a Memoir”. He went from originally trying to write something objective to realizing it was necessarily his personal experience. Even the culture aspect of his story has influenced how I try to foster teamwork. It was an engaging read for me.

Re: DuckDB as the New jq

#56

Post by metadat »

salmo wrote: post #32
digdugdirk wrote: post #31

Earlier quoted context omitted.

Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language. I'm very interested, but not a Linux person, do you know of any good resources for learning the Linux shell as a programming language? [0] https://jqlang.github.io/jq/

I’ll say, I did shell scripting for years from copy/paste, cribbing smarter people, and reading online guides. But I didn’t really understand until I read The Unix Programming Environment by Brian Kernighan and Rob Pike. It’s a very old book and the audience was using dumb terminals. But it made me understand why and how. I think I’ve read every Kernighan book at this point and most he was involved in because he is j…

Really love your comment, so much that I wanted to check out the books you mentioned.

After searching z-lib for "The UNIX Programming Environment", all I found was a janky and grainy PDF. Then I searched archive.org and discovered this high fidelity PDF version:

https://archive.org/details/UnixProgrammingEnviornment

Note: Sadly, the EPUB version is 10x larger (370MB) and is corrupted, not able to be opened / viewed.

Re: DuckDB as the New jq

#57

Post by jonfw »

My current team produces a CLI binary that is available on every build system and everybody's dev machines

Whenever we're writing automation, if the code is nontrivial, or if it starts to include dependencies, we move the code into the CLI tool.

The reason we like this is that we don't want to have to version control tools like duckdb across every dev machine and every build system that might run this script. We build and version control a single binary and it makes life simple.

Re: DuckDB as the New jq

#58

Post by jonfw »

bdcravens wrote: post #26
JeremyNT wrote: post #19

I have a lot of trouble understanding the benefits of this versus just working with json with a programming language. It seems like you're adding another layer of abstraction versus just dealing with a normal hashmap-like data structure in your language of choice. If you want to work with it interactively, you could use a notebook or REPL.

Pipelining CLI commands or bash scripts. From a security perspective, it may be preferable to not ship with a runtime.

Use a compiled language like golang if you don't want to ship with a runtime.

If you're willing to ship w/ bash then I don't understand the opposition to JS. Either tool puts you in a scenario where somebody who can exec into your env can do whatever they want

Re: DuckDB as the New jq

#59

Post by pphysch »

jimbokun wrote: post #23
xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

The Unix philosophy continues to pass the test of time.

Yes and no. Many UNIX philosophy proponents are abhorred by powerful binaries like jq and awk.

Re: DuckDB as the New jq

#60

Post by mdaniel »

philsnow wrote: post #14
HellsMaddy wrote: post #8

Jq tip: Instead of `sort_by(.count) | reverse`, you can do `sort_by(-.count)`

only if you're sure that .count is never null: $ echo '[{"a": {"count": null}}]' | jq -c 'sort_by(-.count)' jq: error (at :1): null (null) cannot be negated $ echo '[{"a": {"count": null}}]' | jq -c 'sort_by(.count) | reverse' [{"a":{"count":null}}]

this whole thread is like nerd sniping me :-D but I felt compelled to draw attention to jq's coalesce operator because I only recently learned about it and searching for the word "coalesce" in the man page is pfffft (it's official name is "Alternative operator", with alternative being "for false and null")

  $ echo '[{"a": {"count": null}}]' | jq -c 'sort_by(-(.count//0))'
  [{"a":{"count":null}}]
Post reply on HN