Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#101

Post by gregjor »

MayeulC wrote: post #94
rustyf wrote: post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

I agree that this is something missing on classic UNIX shells: typed output streams. I had this discussion a while back on HN, though I can't find it ATM (I wish there was a comment history search function). I am far from the first one who thought of that, and there are a few implementations of this idea. Searching for that comment, I came across relevant stories: https://news.ycombinator.com/item?id=27535357 https:/…

Typed output streams aren’t missing from Unix. The authors of Unix clearly describe, repeatedly, that Unix tools use plain text as the common format, and that everything should look like a file.

You’re right that standard Unix tools don’t have a concept of types in streams, but that decision got made deliberately. Types and formats got left as output details.

Analogously my Macbook Air doesn’t have a fan, by design, not by accidental omission.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#102

Post by kbouck »

didgetmaster wrote: post #31
mcdonje wrote: post #12

While I get what they mean by "tabular JSON", that's a bit of a misnomer.

I was wondering what they meant by that term as well. Does every JSON document in the array have to have the exact same structure (including the ordering of key-value pairs)? What happens if row #1000 introduces a new key-value pair not seen before? What if the value for a key is an array?

It generally means an sequence/array of flat json objects, each key represents a column name, and the value representing the row value for that column. Nested json can also be "flattened".

> "... ordering of key-value pairs"

order of appearance of key-value pairs in input does not matter.

> "... if row #100 introduces a new key-value pair"

this is sparse data. miller handles this with the "unsparsify" verb:

    $ cat in.json
    { "a": 1, "b": 2 }
    { "a": 3, "b": 4 }
    { "a": 5, "b": 6, "c": 7 }
without unsparsify:

    $ cat in.json | mlr --j2p cat
    a b
    1 2
    3 4

    a b c
    5 6 7
with unsparsify:

    $ cat in.json | mlr --j2p unsparsify then cat
    a b c
    1 2 -
    3 4 -
    5 6 7
unsparsify can also set default values:

    $ cat in.json | mlr --j2p unsparsify --fill-with 0 then cat
    a b c
    1 2 0
    3 4 0
    5 6 7
> "... What if the value for a key is an array?"

Array value treatment seems to depend on output format. for output types that can represent arrays, they are preserved:

    $ cat in-array.json
    { "a": [0,1,2], "b": [5,6,7] }
    { "a": [3,4,5], "b": [8,9,0] }

    $ cat in-array.json | mlr --jsonl cat
    {"a": [0, 1, 2], "b": [5, 6, 7]}
    {"a": [3, 4, 5], "b": [8, 9, 0]}
for formats like csv/fixed-width, arrays are flattened into columns, one for each array element:

    $ cat in-array.json | mlr --j2p cat
    a.1 a.2 a.3 b.1 b.2 b.3
    0   1   2   5   6   7
    3   4   5   8   9   0
flatten separator can also be set:

    $ cat in-array.json | mlr --j2p --flatsep _ cat
    a_1 a_2 a_3 b_1 b_2 b_3
    0   1   2   5   6   7
    3   4   5   8   9   0

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#103

Post by damascus »

account-5 wrote: post #52
networked wrote: post #40

Earlier quoted context omitted.

I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools .

Thanks for this, good job. I always mean to do something similar but just end up bookmarking HN threads I then never look at.

So chatgpt and I made this: https://github.com/erasei/randombookmark Load in developer mode and when you click the icon it will open a new tab with a random bookmark.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#104

Post by MayeulC »

gregjor wrote: post #101
MayeulC wrote: post #94

Earlier quoted context omitted.

I agree that this is something missing on classic UNIX shells: typed output streams. I had this discussion a while back on HN, though I can't find it ATM (I wish there was a comment history search function). I am far from the first one who thought of that, and there are a few implementations of this idea. Searching for that comment, I came across relevant stories: https://news.ycombinator.com/item?id=27535357 https:/…

Typed output streams aren’t missing from Unix. The authors of Unix clearly describe, repeatedly, that Unix tools use plain text as the common format, and that everything should look like a file. You’re right that standard Unix tools don’t have a concept of types in streams, but that decision got made deliberately. Types and formats got left as output details. Analogously my Macbook Air doesn’t have a fan, by design,…

Right, that was an unfortunate choice of words. It's not missing, but leads to (IMO) a proliferation of under-specified text-based data and stream exchange formats.

Having human-readable text as the lowest common denominator is a laudable goal. Shell scripting would however probably be improved if most tools offered alternative typed streams, or something similar. I am not convinced Powershell's approach is the best, but their approach is at least interesting.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#105

Post by mdaniel »

MayeulC wrote: post #94
rustyf wrote: post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

I agree that this is something missing on classic UNIX shells: typed output streams. I had this discussion a while back on HN, though I can't find it ATM (I wish there was a comment history search function). I am far from the first one who thought of that, and there are a few implementations of this idea. Searching for that comment, I came across relevant stories: https://news.ycombinator.com/item?id=27535357 https:/…

Algolia supports searching comments, and while I don't know the "facet" syntax to pin it to your username, thankfully it's a pretty distinct word and https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... coughed up https://news.ycombinator.com/item?id=26797583 from 2 years ago. Is that the one?

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#106

Post by WinstonSmith84 »

This is a cool tool. And it works well with GPT4. I asked ChatGPT4 the following: ``` I'd like to use Miller from John Kerl on the CSV called test.csv. It has 3 columns, Name, City and State. I'd like to filter City where it's either Chicago or Boston. I want to remove the State column and add a new column called NewColumn. This new column shall be filled with the value of the Name column appended with the letter `_new`. Finally the output shall be saved into the file called myfile.csv. Write the command to do that ```

And it replied with

mlr --csv filter '$City == "Chicago" || $City == "Boston"' then cut -x -f State then put '$NewColumn = $Name . "_new"' test.csv > myfile.csv

I'm sure that someone familiar with Miller would write this command faster than writing the text to GPT, but as a newbie I'd have spent much longer. Also each steps is described perfectly

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#107

Post by zamnos »

krick wrote: post #86
zX41ZdbW wrote: post #68

I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...

> curl https://clickhouse.com/ | sh Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.

User agent "sniffing" (I mean, it's right there. It's not exactly subtle.) has been going on since the before the IE6 days. That it now extends to make things easier for us as command line users is... kinda convenient? Another site where I've seen it done to good effect is http://ifconfig.me . Hit that with a web browser and get a page of accompanying information. Hit it with curl, and get back your ip in ascii - not even an extra newline character is returned!

The underlying question is do you trust clickhouse.com or not? You don't have to; I've never met the team or talked to them, and I can't make that decision for you. But whether you go to the site, laboriously find the download page, right click, download a binary, install the deb/rpm, ask your package manager for what files just got installed, then find and run the clickhouse binary, or just let your computer do it for you via a shell script, the end result is the same. Code from clickhouse (and we're sure it was from clickhouse because of TLS) was downloaded to the target machine, and then got run. Does things have to be difficult and annoying in order for you to like it? (Psychology studies say yes, actually.)

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#108

Post by gregjor »

MayeulC wrote: post #104
gregjor wrote: post #101

Earlier quoted context omitted.

Typed output streams aren’t missing from Unix. The authors of Unix clearly describe, repeatedly, that Unix tools use plain text as the common format, and that everything should look like a file. You’re right that standard Unix tools don’t have a concept of types in streams, but that decision got made deliberately. Types and formats got left as output details. Analogously my Macbook Air doesn’t have a fan, by design,…

Right, that was an unfortunate choice of words. It's not missing, but leads to (IMO) a proliferation of under-specified text-based data and stream exchange formats. Having human-readable text as the lowest common denominator is a laudable goal. Shell scripting would however probably be improved if most tools offered alternative typed streams, or something similar. I am not convinced Powershell's approach is the best,…

Picking nits, but the decision to use plain text as the common data format in Unix does not rule out structured text (e.g. CSV, TSV, XML). Nor does it imply “human-readable.”

The original decision was about not proliferating specialized or proprietary binary formats, which was more of a norm back in the ‘70s than today. The goal was to make small single-purpose tools that communicated through a common interface (files) in a standard format (plain text). Unix succeeded and continues to succeed at that.

Nothing about those design decisions precludes tools using binary formats under Unix — image processing, for example. It just precludes using standard text-oriented tools on those formats.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#109

Post by MayeulC »

mdaniel wrote: post #105
MayeulC wrote: post #94

Earlier quoted context omitted.

I agree that this is something missing on classic UNIX shells: typed output streams. I had this discussion a while back on HN, though I can't find it ATM (I wish there was a comment history search function). I am far from the first one who thought of that, and there are a few implementations of this idea. Searching for that comment, I came across relevant stories: https://news.ycombinator.com/item?id=27535357 https:/…

Algolia supports searching comments, and while I don't know the "facet" syntax to pin it to your username, thankfully it's a pretty distinct word and https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... coughed up https://news.ycombinator.com/item?id=26797583 from 2 years ago. Is that the one?

Ah, interesting feature I didn't know about, thank you.

It's one of the parents in that comment chain (top comment on the story): https://news.ycombinator.com/item?id=26791597

Post reply on HN