Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#91

Post by maximilianroos »

krick wrote: post #86
zX41ZdbW wrote: post #68

I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...

> curl https://clickhouse.com/ | sh Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.

Why be so cantankerous?

    curl https://clickhouse.com/ > install.sh
    cat install

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#92

Post by dmix »

password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

Anyone else find jq really hard to use? I found the syntax/interface bewildering at times. Although not as bad as awk.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#93

Post by cerved »

krick wrote: post #86
zX41ZdbW wrote: post #68

I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...

> curl https://clickhouse.com/ | sh Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.

pipe it though less

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#94

Post by MayeulC »

rustyf wrote: post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

I agree that this is something missing on classic UNIX shells: typed output streams. I had this discussion a while back on HN, though I can't find it ATM (I wish there was a comment history search function). I am far from the first one who thought of that, and there are a few implementations of this idea.

Searching for that comment, I came across relevant stories:

https://news.ycombinator.com/item?id=27535357

https://news.ycombinator.com/item?id=25225113

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#95

Post by avodonosov »

coldtea wrote: post #64
avodonosov wrote: post #46

Earlier quoted context omitted.

csvsql is the first google result for "sql in command line for csv" https://towardsdatascience.com/analyze-csvs-with-sql-in-comm...

Yes, I know. It's not that there isn't several ways to do it, it's that it's not really a good fit for the command line, except in the "I want to reuse SQL that I already know". The problem isn't in having a way to use SQL to query the data from the command line, it's that SQL is long winded and with syntax not really fit in a traditional shell pipeline.

I submitted a ticket: https://github.com/johnkerl/miller/issues/1235

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#96

Post by pxc »

shortrounddev wrote: post #43
hobs wrote: post #19

Earlier quoted context omitted.

While powershell has gotten a lot faster in the most recent versions its still pretty slow for anything involving computation; most of the type my equivalent python code beats the pants off of it. The expressiveness is nice, but oftentimes modules won't support it or require weird ways of using the data to get the performance you want (mostly by dropping out of the pipe.) The choices around Format- vs Out- vs Convert…

My philosophy about any shell language is that if performance is a concern, then you should probably use a real programming language for it. Using shell scripting to handle batch processing tasks just creates dependencies on unmaintainable code.

It's also inhibited by typical corporate crapware, but with a pretty barebones/vanilla config, in PowerShell takes a full four to ten seconds just to open a new session on my work computer.

PowerShell isn't even fast enough for basic interactive usage, never mind batch processing.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#97

Post by password4321 »

dmix wrote: post #92
password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

Anyone else find jq really hard to use? I found the syntax/interface bewildering at times. Although not as bad as awk.

I'm pretty sure that's one big reason for all these other tools!

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#99

Post by kbouck »

garciasn wrote: post #14
avodonosov wrote: post #11

How is it better that SQL for these tasks on tabular data? That's the first question I have after reading the title. Haven't read the article. Edited the first sentence. Originally it was "Why it is not called SQL if works on tabular data?". My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. Awk is too powerful, not sure about it.

1. Because it doesn't use SQL syntax. 2. Because it's closer to an amalgamation of the standard shell scripting tools (cut, sort, jq, etc) than it is to a SQL variant.

and miller also has a powerful (and intuitive imo.) dsl should you need to go beyond what the simple command line switches offer.

so, it remains simple and concise for the easy problems, yet can scale up to address the trickier ones as well.

i've used csvkit, sqlite, jq, etc. miller is my favorite tool for data-fu (with sqlite being a close 2nd)

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#100

Post by coldtea »

melx wrote: post #25
coldtea wrote: post #22

Earlier quoted context omitted.

Well, head just reads lines. A CSV row can optionally be multiple lines. Also head doesn't do anything with the data or the format, aside from printing line by line, so doesn't need to know any column names.

head reads CSV row that has multiple lines. Miller is being offered as "like awk, sed, head.." (emphasis on head - mine) and yes it offers more, but it does not behave "like" the *nix tools it refers to.

>head reads CSV row that has multiple lines.

It just reads lines, doesn't know anything about rows.

So if you want, say, the first 100 rows of a csv that has multiline rows, miller can give it, while head can't.

Head will just print the "N" first lines - whether those are 1, 22, 36 or N actual csv rows.

>Miller is being offered as "like awk, sed, head.." (emphasis on head - mine) and yes it offers more, but it does not behave "like" the nix tools it refers to.*

That's the whole idea.

That it behaves in a way more suited to the csv format, and more coherent than 5-6 different text-focused tools.

The claim is not "this is awk, sed, head, sort remade for csv with identical interfaces and behavior" but "this is a tool to work with csv files and do what you'd normally have to jump through hoops to do with awk, sed, head, sort which don't understand csv structure".

Post reply on HN