Live data from Hacker News

Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

github.com

11–20 of 109 posts

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#11
How is it better that SQL for these tasks on tabular data?

That's the first question I have after reading the title. Haven't read the article.

Edited the first sentence. Originally it was "Why it is not called SQL if works on tabular data?".

My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. Awk is too powerful, not sure about it.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#13

How is it better that SQL for these tasks on tabular data? That's the first question I have after reading the title. Haven't read the article. Edited the first sentence. Originally it was "Why it is not called SQL if works on tabular data?". My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. Awk is too powerful, not sure about it.

Because SQL (Structured Query Language) is the language used to access/manipulate the data, not the name for that kind of data. There have been, and are, many databases, which essentially use tabular data, which are not SQL database.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#14

How is it better that SQL for these tasks on tabular data? That's the first question I have after reading the title. Haven't read the article. Edited the first sentence. Originally it was "Why it is not called SQL if works on tabular data?". My point, if one wants cat, sort, sed, join on tabular data, SOL is exactly that. Awk is too powerful, not sure about it.

1. Because it doesn't use SQL syntax.

2. Because it's closer to an amalgamation of the standard shell scripting tools (cut, sort, jq, etc) than it is to a SQL variant.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#15
post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

Ever try this on large files? A lot of PowerShell commands I make like this can take minutes to run when a combination of Linux commands and Awk might take a couple of seconds.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#16
post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

I much prefer this approach as you only have to parse stuff once during the processing pipeline rather than at every step.

Unfortunately after working with it for a few years I utterly despise PowerShell for many other reasons.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#17
post #16
post #10

Even without aliases, I still prefer PowerShell on Windows. Once you've turned the text into an object, you can pipeline it to hell. Get-Content .\example.csv | ConvertFrom-Csv | Where-Object -Property color -eq red

I much prefer this approach as you only have to parse stuff once during the processing pipeline rather than at every step. Unfortunately after working with it for a few years I utterly despise PowerShell for many other reasons.

Can you share some of the reasons you don't like it?

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#18
post #17
post #16

Earlier quoted context omitted.

I much prefer this approach as you only have to parse stuff once during the processing pipeline rather than at every step. Unfortunately after working with it for a few years I utterly despise PowerShell for many other reasons.

Can you share some of the reasons you don't like it?

Yes: memory ceiling is huge, WinRM is buggy and unreliable, performance is "variable", totally inconsistent handling of dates and times, being .Net it's UTF-16 internally so streams are painful to work with, escape characters argh, variable scoping is horrible, most of the API you have to hit to get stuff done is wrapping ancient bits of weird win32 and DCOM and WMI thus has inconsistent error handling and ability to actually do what it is told.

I've probably forgotten a few things.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#19
post #17
post #16

Earlier quoted context omitted.

I much prefer this approach as you only have to parse stuff once during the processing pipeline rather than at every step. Unfortunately after working with it for a few years I utterly despise PowerShell for many other reasons.

Can you share some of the reasons you don't like it?

While powershell has gotten a lot faster in the most recent versions its still pretty slow for anything involving computation; most of the type my equivalent python code beats the pants off of it.

The expressiveness is nice, but oftentimes modules won't support it or require weird ways of using the data to get the performance you want (mostly by dropping out of the pipe.)

The choices around Format- vs Out- vs Convert- are Very Confusing for new people and the "object in a shell but also text sometimes" way of displaying things is weird and until recently things like -NoTypeInformation or managing encoding in files was just pointlessly weird.

The module support and package management is still entirely in the stone ages and I regularly see people patching in C# in strings to get the behavior they want.

"Larger" modules tend to get Really Slow - the azure modules especially are just an example of how not to do it.

The way it automatically unwraps collections is cool, but gets weird when you have a 1 vs many option for your output, and you might find yourself defensively casting things to lists or w/e.

The typing system in general is nice to get started on but declaring a type does not fix it, so assignment can just break your entire world.

There's still a lot to love about the language when you are getting things done in a windows environment its great to glue together the various pieces of the system but I find the code "breaks" more often than equivalent python code.

Re: Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON

#20
Discussion of a similar tool last month:

yq: command-line YAML, JSON, XML, CSV and properties processor

https://news.ycombinator.com/item?id=34656022

Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

Post reply on HN