Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

Post reply on HN
Print view View original

61–70 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#61

Post by samstave »

account-5 wrote: post #52
networked wrote: post #40

Earlier quoted context omitted.

I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools .

Thanks for this, good job. I always mean to do something similar but just end up bookmarking HN threads I then never look at.

Guilty as charged... I have so many bookmarks I forget to look at...

APPlet Idea: an app that picks a random bookmark from your saves and sends you a reminder to click it and read it... and schedule when you want to see it - like every 7AM send me a link from my bookmarks. Set an alarm for 15 minutes to get me back on schedule.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#63

Post by mdaniel »

nextaccountic wrote: post #58
password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

the obvious missing reference is jq https://github.com/stedolan/jq

I've been getting a lot of mileage out of https://github.com/itchyny/gojq#readme recently due to two things: its vastly superior error messages and the (regrettably verbose) `--yaml-input` option

I also have https://github.com/01mf02/jaq#readme installed but just haven't needed it

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#64

Post by coldtea »

avodonosov wrote: post #46
coldtea wrote: post #27

Earlier quoted context omitted.

> How is it better that SQL for these tasks on tabular data? It has far better handling of a CSV/TSV file on the command line directly and is compasable in shell pipelines. > My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. SQL is a language for data in the form of tables in relational databases. While it can do sorting or joining or some changes, it is meant for a different domain th…

csvsql is the first google result for "sql in command line for csv" https://towardsdatascience.com/analyze-csvs-with-sql-in-comm...

Yes, I know. It's not that there isn't several ways to do it, it's that it's not really a good fit for the command line, except in the "I want to reuse SQL that I already know".

The problem isn't in having a way to use SQL to query the data from the command line, it's that SQL is long winded and with syntax not really fit in a traditional shell pipeline.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#65

Post by horseRad »

networked wrote: post #40
password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools .

You should include qsv (https://github.com/jqnatividad/qsv) and goawk (https://github.com/benhoyt/goawk) into the CSV section! Goawk got a csv/tsv mode and qsv is loaded with features being developed.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#66

Post by gabrielsroka »

2h wrote: post #5
carb wrote: post #4

This is nice. I use `column` for pretty printing CSV/TSV but it fixes two tiny gaps in `sort` (skipping header lines) and `jq` (parsing CSV input. `jq` supports `@csv` for output conversion but not input). $ cat example.csv color,shape,flag,index yellow,triangle,1,11 red,square,1,15 red,circle,1,16 red,square,0,48 purple,triangle,0,51 red,square,0,77 # pretty printing $ column -ts',' example.csv color shape flag inde…

> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv

I thought the Go code looked way too complex and Python would be simpler. Yes and no.

  import csv
  
  filename = 'example.csv'
  sort_by = 'index'
  reverse = True
  
  with open(filename) as f:
      lines = [d for d in csv.DictReader(f)]
  
  for line in lines:
      line['index'] = int(line['index'])
  lines.sort(key=lambda line: line[sort_by], reverse=reverse)
  print(','.join(lines[0].keys()))
  for line in lines:
      print(','.join(str(v) for v in line.values()))

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#67

Post by hobs »

shortrounddev wrote: post #43
hobs wrote: post #19

Earlier quoted context omitted.

While powershell has gotten a lot faster in the most recent versions its still pretty slow for anything involving computation; most of the type my equivalent python code beats the pants off of it. The expressiveness is nice, but oftentimes modules won't support it or require weird ways of using the data to get the performance you want (mostly by dropping out of the pipe.) The choices around Format- vs Out- vs Convert…

My philosophy about any shell language is that if performance is a concern, then you should probably use a real programming language for it. Using shell scripting to handle batch processing tasks just creates dependencies on unmaintainable code.

Simple tools with simple rules will outlast most of the code we'll all build. Picking things like grep, awk, and sed means your knowledge will be widely applicable going forward, and many people caring about their performance both backwards and forwards in time means your shell can be pretty fast.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#68

Post by zX41ZdbW »

I recommend using clickhouse-local[1] for these tasks.

It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2].

[1] https://clickhouse.com/blog/extracting-converting-querying-l...

[2] https://colab.research.google.com/github/dcmoura/spyql/blob/...

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#69

Post by elesiuta »

networked wrote: post #40
password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools .

Thank you! That list is perfect since I don't use these tools often enough to remember them and some are hard to find again without knowing exactly what to search for.

I also discovered pawk on it which looks interesting, I made a somewhat related tool [1] which is probably out of scope for your list, but can be used in some of the same ways [2] and may of of interest nonetheless.

[1] https://github.com/elesiuta/pyxargs

[2] cat /etc/hosts | pyxargs -d \n --im json --pre "d={}" --post "print(dumps(d))" --py "d['{}'.split()[0]] = '{}'.split()[1]"

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#70

Post by elesiuta »

password4321 wrote: post #20

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

Another similar discussion last year where Miller was mentioned:

New(ish) command line tools

https://news.ycombinator.com/item?id=31009313

Post reply on HN