Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

1–10 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#4
This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input).

  $ cat example.csv 
  color,shape,flag,index
  yellow,triangle,1,11
  red,square,1,15
  red,circle,1,16
  red,square,0,48
  purple,triangle,0,51
  red,square,0,77
  
  # pretty printing
  $ column -ts',' example.csv 
  color   shape     flag  index
  yellow  triangle  1     11
  red     square    1     15
  red     circle    1     16
  red     square    0     48
  purple  triangle  0     51
  red     square    0     77
  
  # sorting with skipped headers is a mess. 
  $ (head -n 1 example.csv && tail -n +2 example.csv | sort -r -k4 -t',') | column -ts','
  color   shape     flag  index
  red     square    0     77
  purple  triangle  0     51
  red     square    0     48
  red     circle    1     16
  red     square    1     15
  yellow  triangle  1     11

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#5
post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

> sorting with skipped headers is a mess

I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example:

https://go.dev/play/p/9bASZ97lLWv

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#6
+1 for easy install (`dnf install miller` on my Fedora).

But seems like it cannot handle a simple use case: CSV without header.

$ mlr --csv head -n 20 pp-2002.csv

mlr: unacceptable empty CSV key at file "pp-2002.csv" line 1.

You have to explicitly pass it (FYI `implicit-csv-header` is terrible arg name)

$ mlr --csv --implicit-csv-header head -n 20 pp-2002.csv

While `head` obliges rightly

$ head -n 20 pp-2002.csv

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#7
post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

Another option (based on https://unix.stackexchange.com/q/11856/109046):

    (sed -u '1q' ; sort -r -k4 -t',') 

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#8
post #5
post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv

It's a matter of perspective.

I like programming languages, but I think they can cripple someone actually learning Unix!

At the end of the day, you should just use whatever tools make you the most productive most quickly.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#9
Great tool.

BUT, leaks memory like crazy.

Despite documentation stating the verbs are fully streaming.

> Fully streaming verbs > These don't retain any state from one record to the next. They are memory-friendly, and they don't wait for end of input to produce their output.

https://miller.readthedocs.io/en/6.7.0/streaming-and-memory/...

Post reply on HN