71–80 of 109 posts
I've been using chat-GPT to work with these types of files now.
Earlier quoted context omitted.
the obvious missing reference is jq https://github.com/stedolan/jq
I've been getting a lot of mileage out of https://github.com/itchyny/gojq#readme recently due to two things: its vastly superior error messages and the (regrettably verbose) `--yaml-input` option I also have https://github.com/01mf02/jaq#readme installed but just haven't needed it
Previous discussion: https://news.ycombinator.com/item?id=28298729 (273 points | Aug 25, 2021 | 66 comments)
Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON - https://news.ycombinator.com/item?id=29651871 - Dec 2021 (33 comments)
Miller CLI – Like Awk, sed, cut, join, and sort for CSV, TSV and JSON - https://news.ycombinator.com/item?id=28298729 - Aug 2021 (66 comments)
Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions - https://news.ycombinator.com/item?id=13751389 - Feb 2017 (20 comments)
Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV - https://news.ycombinator.com/item?id=10066742 - Aug 2015 (76 comments)
I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.
the obvious missing reference is jq https://github.com/stedolan/jq
Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red
No thank you. I appreciate the power, speed, simplicity and flexibility of UNIX/GNU style text tools. I-Also-Don't-Want-To-Be-Locked-Into-This-Ridiculous-Syntax-Nightmare.
ipcsv example.csv | ? color -eq red
color shape flag index
----- ----- ---- -----
red square 1 15
red circle 1 16
red square 0 48
red square 0 7Earlier quoted context omitted.
No thank you. I appreciate the power, speed, simplicity and flexibility of UNIX/GNU style text tools. I-Also-Don't-Want-To-Be-Locked-Into-This-Ridiculous-Syntax-Nightmare.
It's funny to me that people hate how verbose posh is when bash syntax looks like my cat walked over my keyboard
I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
or sqlite for CSV/TSV, haven't tried it for json.
I recommend using clickhouse-local[1] for these tasks. It does SQL; it supports all imaginable data formats, streaming processing, and connecting to external data sources. It also outperforms every other tool[2]. [1] https://clickhouse.com/blog/extracting-converting-querying-l... [2] https://colab.research.google.com/github/dcmoura/spyql/blob/...
or sqlite for CSV/TSV, haven't tried it for json.
select a,b,c from '*.jsonl.gz'
Earlier quoted context omitted.
> sorting with skipped headers is a mess I like these command line tools, but I think they can cripple someone actually learning programming language. For example, here is a short program that does your last example: https://go.dev/play/p/9bASZ97lLWv
I thought the Go code looked way too complex and Python would be simpler. Yes and no. import csv filename = 'example.csv' sort_by = 'index' reverse = True with open(filename) as f: lines = [d for d in csv.DictReader(f)] for line in lines: line['index'] = int(line['index']) lines.sort(key=lambda line: line[sort_by], reverse=reverse) print(','.join(lines[0].keys())) for line in lines: print(','.join(str(v) for v in lin…