Live data from Hacker News

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

johnkerl.org

41–50 of 81 posts

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#41
post #19

Can anyone suggest something similar for Windows?

PowerShell can handle CSV files natively. Here are the equivalent commands from the article: Import-Csv example.csv | Select-Object year, price | Export-Csv -NoTypeInformation example-cut.csv Import-Csv example.csv | Sort-Object year | Export-Csv -NoTypeInformation example-sorted.csv Import-Csv example.csv | Where-Object year -eq 1999 | Export-Csv -NoTypeInformation example-filtered.csv PowerShell can also deal with…

This neatly really illustrates why objects matter.

As cool as miller is, jq is also decidedly cool, but for json. Both claim that they are "like" awk, sed, cut, paste etc, but for $Format, where $Format is csv/tsv and json, respectively.

They both deal with much the same tasks, and there is considerable overlap.

What your PowerShell examples do not say, is how - after Import-Csv - you are no longer constrained to tools written for csv's (or json or ...). At that point the records of the csv file have become objects in the PowerShell type system (which is an extension of the .NET type system).

Another point about the corresponding functionality in PowerShell is how you do not need to convert back to csv, json or whatever. Your Export-Csv commands in your examples are only needed if you want to finally save the results as csv. Otherwise you would just keep the objects in a variable or flowing through the pipeline/script.

So TFAs 2nd example could be written

    ipcsv flins.csv | group county -n
If the data had been in json, one would write

    cat flins.json | convertfrom-json | group county -n
Which also means that data can be export through any cmdlet that exports or converts a stream of objects: Export-Csv, ConvertTo-Csv, ConvertTo-Html, ConvertTo-Json, ConvertTo-Xml, ...

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#42
post #11

There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.

Miller looks very useful, and so does q. Thanks to the op and to you for the introductions.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#43

I love csvkit[0] for this purpose [0] https://github.com/onyxfish/csvkit

csvkit is excellent. I like `csvlook` for display csv files in the terminal. Can anyone recommend an easy way to convert csv to xls on Linux? I know that libreoffice and gnumeric can do it in headless mode, but it seems terrible overkill.

A good way to find those kinds of conversion tools is to search using the x2y pattern. For example, searching "csv2xls" returns:

https://github.com/ferkulat/csv2xls

https://github.com/alexprengere/csv2xls

http://www.bitools.eu/csv2xls/

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#44
post #11

There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.

does it bring any advantages over https://metacpan.org/pod/DBD::CSV ? (not being perl is not an advantage).

Well, DBD::CSV is just a driver for DBI, not a command that is "unix pipeline friendly"... So there's that.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#45
post #12

This is lovely. Portable C, awesome.

Not totally portable. They have built some ctags option dependencies that I'm going to have to futz with in order to get the makefile to run. 12:58 shephard:c shephard$ make all ctags -R . ctags: illegal option -- R usage: ctags [-BFadtuwvx] [-f tagsfile] file ... make: *** [tags] Error 1 [Edit - No "-R" option for ctags in OS X 10.8.5, or in OpenBSD Current - http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/ma…

yes, there is an issue open. I have . in my path. will post a diff.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#47

It looks like this tool doesn't support semicolon-separated CSV files, quotes, or numbers using a comma as decimal separator. Too bad. It's a pity that the CSV file format is so fragmented. It's all too common that a CSV file written by tool A can't be read by tool B. Only very few tools do it right. For example, I think the ruby CSV module has a heuristic to automatically detect the "style" of CSV files, which is pr…

semicolons etc you get with FS RS OS etc. number commas I can probably implement with locale; i'll look. and quotes are definitely to do. :)

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#48

It looks like this tool doesn't support semicolon-separated CSV files, quotes, or numbers using a comma as decimal separator. Too bad. It's a pity that the CSV file format is so fragmented. It's all too common that a CSV file written by tool A can't be read by tool B. Only very few tools do it right. For example, I think the ruby CSV module has a heuristic to automatically detect the "style" of CSV files, which is pr…

[deleted]

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#49
post #27

Nothing wrong with this approach I suppose but wouldn't most people be better off to just use a database for this sort of thing?

Depends on your familiarity with the unix toolset vs use cases, an amusing (if somewhat contrived) example of this: http://aadrake.com/command-line-tools-can-be-235x-faster-tha... Most CSV analysis is static, not something ongoing with inserts/updates/etc, so using a combination of unix tools to handle the data processing seems like a sufficient solution for many scenarios.

+1 and nice link :)

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#50
post #12

This is lovely. Portable C, awesome.

He says he used GNU C99, how is that portable C? http://johnkerl.org/miller/doc/whyc.html#C_is_old-school

my aim is portability within a 21st century context ... I need to find a succinct way to say that. the word "portable" is the wrong choice; sounds like it would compile on everything ever which I don't aim for.
Post reply on HN