Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern Linux Tools vs. Unix Classics: Which Would I Choose?

meetryanflowers.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#41

Post by 1vuio0pswjnm7 »

"Those tried and true commands we were referring to? None other than the usual awk sed cut grep and of course the Unix pipe | to glue them all together. Really, why use a JSON parsing program that only could only do one function (parse JSON) when I could use a combination of tools that, when piped together, could do far more?"

IMHO, drinking the UNIX Kool-Aid means not only using coreutils and BSD userlands but also using the language in which almost all of those programs are written: C. For me, that means gcc and binutils are amongst the "tried and true commands". Also among them is flex. These are found on all the UNIX varieties I use, usually because they are used in compiling the OS. As such, no special installation is needed.

When I looked at jq source code in 2013, I noticed it used flex and possibly yacc/bison. No idea if it still does.

Using UNIX text processing utilities to manipulate JSON is easy enough. However if I am repeatedly processing JSON from the same source, e.g., YouTube, then I use flex instead of sed, etc. It's faster.

jq uses flex in the creation of a language interpreter intended^1 to process any JSON. I use flex not to create a language interpeter but to process only JSON from a single source. The blog author uses shell script to process JSON from a single source.^2 I think of the use I make of flex as like a compiled shell script. It's faster.

The blog author states than jq is specific to one type of text processing input: JSON. I write a utility that is specific to one source of JSON.

1. Sometimes it's not used as intended, e.g., https://github.com/makenowjust/bf.jq

2. I also used flex to make simple utility to reformat JSON from any source so it's easer to read and process with line-oriented UNIX utilities. Unlike jq and other JSON reformatters it does not require 100% correct JSON; e.g., it can accept JSON that is mixed in with HTML which I find is quite common in today's web pages.

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#42

Post by zoom6628 »

Still use awk since I learnt it in 1991 ans only use the others when I can make awk do all I need so that means leaning on python for more complex logic when required. Tried ha a few times and it did my head in too (yes I'm a lazy idiot obviously - guilty as charged because if I wasn't lazy I would write a JDON parser I could actually use!).

I naturally gravitate to the simplest solution with the most well known and proven tools. Not fan of boiling oceans or immersing in obscurity when there is a deadline to meet.

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#43

Post by ninjin »

mongol wrote: post #19

I recall a tool that rewrites json to a dot notation that is easily grep-able. It prepended each value so all the parents were in front, something like a.b.c=d But I have forgot the name.. Edit: was already mentioned in the thread! Gron

There is also json2tsv [1] that follows a similar philosophy and I have had some fun combining it with awk(1) recently for database ingestion.

[1]: https://www.codemadness.org/json2tsv.html

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#44

Post by bigstrat2003 »

hnbear wrote: post #20

jq is definitely tough to learn, I can never remember it. But, the whole argument against jq as a unitasker not worth learning and traditional unix tools being better is weird. Traditionally Unix tool mentality was “do one thing only and do it well” then pipe it together. jq fits perfectly into the Unix toolset.

Also, Alton Brown doesn't advise against unitaskers because he's some kind of hater. The reason you don't have unitaskers in the kitchen is because they take up physical space. You have to decide which tools are worth the space they take up, and it's hard to justify that for a tool you will only use once in a while. That's not a concern with computers any more, so the reasoning doesn't apply here.

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#45

Post by INTPenis »

This post reads so funny, "cheating"? Just use the right tool for the job buddy.

I just recently built jq into a container image so I could parse json inside of it. Installing jq isn't a hassle if you do it once.

Stop focusing on culture, kool-aid, and cheating and just get the job done.

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#46

Post by eternityforest »

defrost wrote: post #10
umvi wrote: post #4

I too can never remember jq syntax when I need to. I usually just end up writing a Python script to extract the part of the JSON I need.

If JQ is too much, see GRON &| Miller gron transforms JSON into discrete assignments to make it easier to grep for what you want https://github.com/tomnomnom/gron Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON https://github.com/johnkerl/miller

Really cool but... Python exists. If I am gonna do anything with JSON that is probably well into "Time to use a full programming language" territory

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#48

Post by rusk »

ludwigvan wrote: post #47

Using awk/sed to parse json seems to be using the wrong tool for the job. As an alternative to jq with easier to remember syntax, see https://fx.wtf/ Recent discussion: https://news.ycombinator.com/item?id=37567009

Personally I prefer ‘python -m json.tool’

I resent the insular tone of jq. If I’m going to be processing the data in any way I’d prefer to make the leap into a familiar environment rather than kid myself I’m querying the doc like SQL

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#49

Post by defrost »

eternityforest wrote: post #46
defrost wrote: post #10

Earlier quoted context omitted.

If JQ is too much, see GRON &| Miller gron transforms JSON into discrete assignments to make it easier to grep for what you want https://github.com/tomnomnom/gron Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON https://github.com/johnkerl/miller

Really cool but... Python exists. If I am gonna do anything with JSON that is probably well into "Time to use a full programming language" territory

FWiW my preference for JSON related data transforms and driven computation is almost always JQ - it is a Turing-complete interpreter language with JSON iterator as the first-citizen object.

I suggested the above for those that can't justify the time it takes to make JQ perform well for whatever relatively minor needs they have.

JQ is back under active development and there are a number of optimised | Go | Rust spinoff github projects that'll likely do some interesting work now that the mainline is starting to release again.

Re: Modern Linux Tools vs. Unix Classics: Which Would I Choose?

#50

Post by sbuk »

zgs wrote: post #11

The title should read modern GNU tools. Linux is a kernel and supplies nothing useful for command line lovers. Nonetheless, the more modern tools are generally better, faster and more feature rich. I'd pick them over the older versions unless there was a compelling reason not to. Yes, I've used both. For almost forty years now.

'Modern tools vs. GNU coreutils on the Unix and Unix-like command line' would be more apt (no pun intended). Remember, GNU is Not Unix...
Post reply on HN