Earlier quoted context omitted.
or sqlite for CSV/TSV, haven't tried it for json.
How does it look from command-line for streaming processing of CSV/TSV?
you can pipe to it
Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
81–90 of 109 posts
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#82Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#83Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.
Benthos is a really cool tool when you want to take this idea to the next level. It can do the usual text/csv/json/yaml mangling you'd do with awk or jq (since it includes a version of those in addition to its own processing langauge) but it also has decoders and encoders for a bunch of different binary formats like protobuf or Avro. And in addition to stdin/out and files it can talk to Kafka, MQTT, HTTP, ES and a bunch of other stuff. I was able to put together a log processor that consumes from Kafka, does a bunch of field mangling and then ingests into Elasticsearch in a couple dozen lines of yaml.
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#84Earlier quoted context omitted.
How does it look from command-line for streaming processing of CSV/TSV?
you can pipe to it
Yep:
$ cat foo.tsv
name foo bar
Alice 10 8888
Bob 20 9999
$ cat foo.tsv | sqlite3 -batch \
-cmd ".mode tabs" \
-cmd ".import /dev/stdin x" \
-cmd "select foo from x where bar > 9000;"
20Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#85Earlier quoted context omitted.
> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv
I thought the Go code looked way too complex and Python would be simpler. Yes and no. import csv filename = 'example.csv' sort_by = 'index' reverse = True with open(filename) as f: lines = [d for d in csv.DictReader(f)] for line in lines: line['index'] = int(line['index']) lines.sort(key=lambda line: line[sort_by], reverse=reverse) print(','.join(lines[0].keys())) for line in lines: print(','.join(str(v) for v in lin…
Perhaps a `DictWriter` would simplify things:
import csv
import sys
filename = "example.csv"
sort_by = "index"
reverse = True
with open(filename, newline="") as f:
reader = csv.DictReader(f)
writer = csv.DictWriter(sys.stdout, fieldnames=reader.fieldnames)
writer.writeheader()
writer.writerows(sorted(reader, key=lambda row: int(row[sort_by]), reverse=reverse))Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#86I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
> curl https://clickhouse.com/ | sh
Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#87Earlier quoted context omitted.
I thought the Go code looked way too complex and Python would be simpler. Yes and no. import csv filename = 'example.csv' sort_by = 'index' reverse = True with open(filename) as f: lines = [d for d in csv.DictReader(f)] for line in lines: line['index'] = int(line['index']) lines.sort(key=lambda line: line[sort_by], reverse=reverse) print(','.join(lines[0].keys())) for line in lines: print(','.join(str(v) for v in lin…
Perhaps a `DictWriter` would simplify things: import csv import sys filename = "example.csv" sort_by = "index" reverse = True with open(filename, newline="") as f: reader = csv.DictReader(f) writer = csv.DictWriter(sys.stdout, fieldnames=reader.fieldnames) writer.writeheader() writer.writerows(sorted(reader, key=lambda row: int(row[sort_by]), reverse=reverse))
I thought about that but 1) it seemed like cheating to write to standard out, 2) you're assuming that the column to sort by is an integer whereas I broke that code up a little bit.
But yours has the advantage of being able to support more complex CSVs.
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#88I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
> curl https://clickhouse.com/ | sh Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#89I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
> curl https://clickhouse.com/ | sh Jesus, this is disgusting. I'm not that picky and don't really complain about "... | sh" usually, but at least I took it for granted that I can always look at the script in the browser and assume that is has no actual evil intentions and doesn't rely on some fucking client-header magic to be modified on the fly.
We provide .deb, .rpm, .tgz, Docker, or single-binary, for x86-64, AArch64, for Linux, Mac and FreeBSD.
Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
#90+1 for easy install (`dnf install miller` on my Fedora). But seems like it cannot handle a simple use case: CSV without header. $ mlr --csv head -n 20 pp-2002.csv mlr: unacceptable empty CSV key at file "pp-2002.csv" line 1. You have to explicitly pass it (FYI `implicit-csv-header` is terrible arg name) $ mlr --csv --implicit-csv-header head -n 20 pp-2002.csv While `head` obliges rightly $ head -n 20 pp-2002.csv
you could shorten to:
$ mlr -N --csv head -n 20 pp-2002.csv
-N is a shortcut for --implicit-csv-header and --headerless-csv-output