Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB as the New jq

pgrs.net

Post reply on HN
Print view View original

61–70 of 74 posts

Re: DuckDB as the New jq

#61

Post by snthpy »

Hi,

I very much share your sentiment and I saw a few comments mentioning PRQL so I thought it might be worth bringing up the following:

In order to make working with data at the terminal as easy and fun as possible, some time ago I created pq (prql-query) which leverages DuckDB, DataFusion and PRQL.

Unfortunately I am currently not in a position to maintain it so the repo is archived but if someone wanted to help out and collaborate we could change that.

It doesn't have much in the way of json functions out-of-the-box but in PRQL it's easy to wrap the DuckDB functions for that and with the new PRQL module system it will soon also become possible to share those. If you look through my HN comment history I did provide a JSON example before.

Anyway, you can take a look at the repo here: https://github.com/PRQL/prql-query

If interested, you can get in touch with me via Github or the PRQL Discord. I'm @snth on both.

Re: DuckDB as the New jq

#62

Post by xg15 »

DanielHB wrote: post #54
xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

found out recently that jq can url-encode values, is there anything it _can't_ do?

Finding out whether . is contained in a given array or not, evidently.

(That's not strictly true - you can do it, you just have to bend over backwards for what is essentially the "in" keyword in python, sql, etc. jq has no less than four functions that look like they should do that - in(), has(), contains() and inside(), yet they all do something slightly different)

Re: DuckDB as the New jq

#63

Post by arp242 »

salmo wrote: post #55
pcthrowaway wrote: post #52

Earlier quoted context omitted.

I wasn't aware of the bait and switch at the time I read it, but I did really enjoy the history of how the Unix/Linux ethic came together and evolved over time. Had I heard of The Unix Programming Environment when I read it in 2014 I may have gone with that instead, as I was looking for something more along the lines of a technical handbook rather than a code of ethics.

Yeah and ESR can be revisionist in his history, projecting intention on something organic. He alienated a lot of people over time with this… and other behavior. The book I recommended is both a handbook and a “how to think.” It applies forward to things introduced well after the book. But it also helped me understand why the Byzantine behavior of a tty is what it is. If you are interested in the history from a first…

I felt it was a good dive into "good" programs at a bit of higher level, rather than "here's how to do X". Quite a bit applies to Windows software and other software that never touches Unix as well.

Some bits are better than others, some bits haven't aged too well in the last 20 years, and it's a shame esr has since turned crazy. But I still feel the book holds up reasonably well.

"Bait and switch" certainly seems too strong of an accusation, especially for a book that's available for free.

I do agree that even pre-crazy esr was never on the level of Kernighan in any way.

Re: DuckDB as the New jq

#64

Post by mutant »

hprotagonist wrote: post #3

i've been using simonw's sqlite-utils ( https://sqlite-utils.datasette.io/en/stable/ ) for this sort of thing; given structured json or jsonl, you can throw data at an in-memory sqlite database and query away: https://sqlite-utils.datasette.io/en/stable/cli.html#queryin...

Thank you was going to say this as well, sqlite does this the same as duckdb

Re: DuckDB as the New jq

#67

Post by boneitis »

salmo wrote: post #32
digdugdirk wrote: post #31

Earlier quoted context omitted.

Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language. I'm very interested, but not a Linux person, do you know of any good resources for learning the Linux shell as a programming language? [0] https://jqlang.github.io/jq/

I’ll say, I did shell scripting for years from copy/paste, cribbing smarter people, and reading online guides. But I didn’t really understand until I read The Unix Programming Environment by Brian Kernighan and Rob Pike. It’s a very old book and the audience was using dumb terminals. But it made me understand why and how. I think I’ve read every Kernighan book at this point and most he was involved in because he is j…

Much thanks for this. I've been struggling to grasp what's happening down in the engine room beneath me as I operate a modern Linux environment and only knew that there have been different waves of evolutions over the decades. I instead only see today's pretty deck up top, outside.

I'm only ten pages in, but that's mostly because the format and approach of this book is quickly yielding me tons of conversation fodder for ChatGPT4, where I've been endlessly asking it to clarify or point out holes in my mental model of everything I'm doing in the terminal, and still working out things such as why Backspace, Ctrl+H, Ctrl+/, Ctrl+?, Ctrl+-, and Ctrl+_ all together seem to have some overlap or differences in what's happening, depending on terminal contexts. I always had working notions of raw versus cooked sessions, the line discipline, etc., but I'm finding ways to play with the machinery with silly exercises that I otherwise couldn't have come up with.

For example, I just opened up three terminal windows:

  1) `man ascii`
  2) `nc -lvp 9001 | xxd -c1`
  3) `stty raw -echo; nc -nv 127.0.0.1 9001`
Then, reproducing the whole "Hex" column of the manpage, in order, using my keystrokes. And, observing the multi-byte payloads of some of the other keys. I feel like a kid again :)

Re: DuckDB as the New jq

#68

Post by jb3689 »

xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

Driving that further - I don’t want to have to edit my query for minutes when I am in the shell. I don’t believe that the shell is conducive to complex SQL queries. You could write simpler SQL queries, but then you’re in a space where there are less verbose tools for those simpler tasks.

DuckDB seems stronger for someone who needs to create a scripting library - which also has lots of options and competition - or someone who has a very specific workflow of working with JSON dumps for a huge percentage of their time.

Re: DuckDB as the New jq

#69

Post by michaelcampbell »

mightybyte wrote: post #42
xg15 wrote: post #18

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot m…

Your command line solution doesn't give quite the same result as OP. The final output in OP is sorted by the count field, but your command line incantation doesn't do that. One might respond that all you need to do is add a second "| sort" at the end, but that doesn't quite do it either. That will use string sorting instead of proper numeric sorting. In this example with only three output rows it's not an issue. But…

> I think it would be that SQL is more powerful than ad-hoc jq incantations.

I don't disagree, but only because I have a lot of experience with SQL so I "think" more in those terms. The blog post made perfect sense to me, and I have to go to claude/openai/copilot for whatever I need in `jq`, EVERY TIME. Because I don't use it nearly as often, so I don't have its language internalized.

Readability (and I'd posit to the point of this, power) is more a function of the reader than the code, pg's "blub paradox" notwithstanding.

I didn't know anything about duckdb, but I might give this a shot.

Re: DuckDB as the New jq

#70

Post by michaelcampbell »

pphysch wrote: post #59
jimbokun wrote: post #23

Earlier quoted context omitted.

The Unix philosophy continues to pass the test of time.

Yes and no. Many UNIX philosophy proponents are abhorred by powerful binaries like jq and awk.

Like any religion, there are zealots and "no true Scotsman" arguments a'plenty.
Post reply on HN