Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

Post reply on HN
Print view View original

21–30 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#21

Post by coldtea »

snidane wrote: post #9

Great tool. BUT, leaks memory like crazy. Despite documentation stating the verbs are fully streaming. > Fully streaming verbs > These don't retain any state from one record to the next. They are memory-friendly, and they don't wait for end of input to produce their output. https://miller.readthedocs.io/en/6.7.0/streaming-and-memory/...

>BUT, leaks memory like crazy.

Huh?

a) Isn't it written in Golang, which has a GC? Does it do custom buffer based management?

b) Isn't it supposed to be run on a file and get some output - as opposed to an interactice session? Why would it matter if it leaks, then, and how could it leak, as the memory is returned to the OS when it ends?

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#22

Post by coldtea »

melx wrote: post #6

+1 for easy install (`dnf install miller` on my Fedora). But seems like it cannot handle a simple use case: CSV without header. $ mlr --csv head -n 20 pp-2002.csv mlr: unacceptable empty CSV key at file "pp-2002.csv" line 1. You have to explicitly pass it (FYI `implicit-csv-header` is terrible arg name) $ mlr --csv --implicit-csv-header head -n 20 pp-2002.csv While `head` obliges rightly $ head -n 20 pp-2002.csv

Well, head just reads lines. A CSV row can optionally be multiple lines.

Also head doesn't do anything with the data or the format, aside from printing line by line, so doesn't need to know any column names.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#23

Post by coldtea »

2h wrote: post #5
carb wrote: post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv

The whole point of UNIX userland is to not have to write a custom program for every simple case that just needs recombining some existing basic programs in a pipeline...

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#25

Post by melx »

coldtea wrote: post #22
melx wrote: post #6

+1 for easy install (`dnf install miller` on my Fedora). But seems like it cannot handle a simple use case: CSV without header. $ mlr --csv head -n 20 pp-2002.csv mlr: unacceptable empty CSV key at file "pp-2002.csv" line 1. You have to explicitly pass it (FYI `implicit-csv-header` is terrible arg name) $ mlr --csv --implicit-csv-header head -n 20 pp-2002.csv While `head` obliges rightly $ head -n 20 pp-2002.csv

Well, head just reads lines. A CSV row can optionally be multiple lines. Also head doesn't do anything with the data or the format, aside from printing line by line, so doesn't need to know any column names.

head reads CSV row that has multiple lines.

Miller is being offered as "like awk, sed, head.." (emphasis on head - mine) and yes it offers more, but it does not behave "like" the *nix tools it refers to.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#26

Post by 2h »

coldtea wrote: post #23
2h wrote: post #5

Earlier quoted context omitted.

> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv

The whole point of UNIX userland is to not have to write a custom program for every simple case that just needs recombining some existing basic programs in a pipeline...

> simple case

that's just it though, the last example is not a simple case, hence why the last example is awkward by the commenters own admission. command line tools are fine, but you need to know when to set the hammer down and pick up the chainsaw.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#27

Post by coldtea »

avodonosov wrote: post #11

How is it better that SQL for these tasks on tabular data? That's the first question I have after reading the title. Haven't read the article. Edited the first sentence. Originally it was "Why it is not called SQL if works on tabular data?". My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. Awk is too powerful, not sure about it.

>How is it better that SQL for these tasks on tabular data?

It has far better handling of a CSV/TSV file on the command line directly and is compasable in shell pipelines.

>My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that.

SQL is a language for data in the form of tables in relational databases.

While it can do sorting or joining or some changes, it is meant for a different domain than these tools, which other constraints, other concerns, and other patterns of use...

You don't need to load anything to a db, for starters.

You also normally don't care for involving a DB in order to use in a shell script, or for quick shell exploration.

You also can't mix SQL and regular unix userland in a pipeline (well, with enough effort you can, but it's not something people do or need to do).

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#28

Post by coldtea »

2h wrote: post #26
coldtea wrote: post #23

Earlier quoted context omitted.

The whole point of UNIX userland is to not have to write a custom program for every simple case that just needs recombining some existing basic programs in a pipeline...

> simple case that's just it though, the last example is not a simple case, hence why the last example is awkward by the commenters own admission. command line tools are fine, but you need to know when to set the hammer down and pick up the chainsaw.

>the last example is not a simple case

As far as shell scripting goes, this is hardly anything to write home about. Looks simple enough to me.

It just retains the header by printing the header first as is, and then sorting the lines after the header. It's immediately obvious how to do it to anybody who knows about head and tail.

And with Miller it's even simpler than that, still on the command line...

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#29

Post by 0cf8612b2e1e »

melx wrote: post #25
coldtea wrote: post #22

Earlier quoted context omitted.

Well, head just reads lines. A CSV row can optionally be multiple lines. Also head doesn't do anything with the data or the format, aside from printing line by line, so doesn't need to know any column names.

head reads CSV row that has multiple lines. Miller is being offered as "like awk, sed, head.." (emphasis on head - mine) and yes it offers more, but it does not behave "like" the *nix tools it refers to.

Miller is designed around the idea of structured data. This is a higher bar than naively manipulating text, and the user has to be more deliberate to extract fields. Doing cli manipulation of a csv that contains quoted commas is challenging with the standard tools.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#30

Post by darrenf »

carb wrote: post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

> `jq` supports `@csv` for output conversion but not input

Actually, `jq` can cope with trivial CSV input like your example, - `jq -R 'split(",")'` will turn a CSV into an array of arrays. To then sort it in reverse order by 3rd column and retain the header, the following fell out of my fingers (I'm beyond certain that a more skilled `jq` user than me could improve it):

     jq -R 'split(",")' example.csv | jq -sr '[[.[0]],.[1:]|sort_by(.[4])|reverse|.[]]|.[]|@csv'

    "color","shape","flag","index"
    "red","square","0","77"
    "purple","triangle","0","51"
    "red","square","0","48"
    "red","circle","1","16"
    "red","square","1","15"
    "yellow","triangle","1","11"
NB. there is also an entry in the `jq` cookbook for parsing CSVs into arrays of objects (and keeping numbers as numbers, dealing with nulls, etc) https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-f...
Post reply on HN