Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

johnkerl.org

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#51

Post by gambiter »

jakobegger wrote: post #40

It looks like this tool doesn't support semicolon-separated CSV files, quotes, or numbers using a comma as decimal separator. Too bad. It's a pity that the CSV file format is so fragmented. It's all too common that a CSV file written by tool A can't be read by tool B. Only very few tools do it right. For example, I think the ruby CSV module has a heuristic to automatically detect the "style" of CSV files, which is pr…

CSV stands for 'Comma-Separated Values', not 'Semicolon-Separated Values'.

Seeing as this was downvoted, I'm just pointing out that bemoaning how the CSV format is fragmented and then using it incorrectly is the precise reason it is fragmented. It wasn't originally meant to be extended in so many ways, so there really isn't room to complain when a library doesn't magically figure out your personal delimiter or qualifier.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#52

Post by john_kerl »

ghshephard wrote: post #20
natch wrote: post #12

This is lovely. Portable C, awesome.

Not totally portable. They have built some ctags option dependencies that I'm going to have to futz with in order to get the makefile to run. 12:58 shephard:c shephard$ make all ctags -R . ctags: illegal option -- R usage: ctags [-BFadtuwvx] [-f tagsfile] file ... make: *** [tags] Error 1 [Edit - No "-R" option for ctags in OS X 10.8.5, or in OpenBSD Current - http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/ma…

ctags etc I'll split out. effectively there are make targets to make dev nice, and a smaller set just to make the binary. ctags definitely in the former category. thanks for the feedback!

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#53

Post by pessimizer »

Whenever somebody posts a csv manipulation tool, it's obligatory for everyone else to post their favorite one too: http://neilb.bitbucket.org/csvfix/ is mine; manual at http://neilb.bitbucket.org/csvfix/manual/csvfix16/csvfix.htm...

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#54

Post by functional_test »

loevborg wrote: post #37
michaelmior wrote: post #8

I love csvkit[0] for this purpose [0] https://github.com/onyxfish/csvkit

csvkit is excellent. I like `csvlook` for display csv files in the terminal. Can anyone recommend an easy way to convert csv to xls on Linux? I know that libreoffice and gnumeric can do it in headless mode, but it seems terrible overkill.

As others have mentioned, you can find variations of this around online, but if you wish to roll you own (perhaps to add other features or what have you), you could knock it out in about 20 lines of Python using the built in CSV library and Openpyxl [1].

[1] https://openpyxl.readthedocs.org/en/latest/

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#55

Post by john_kerl »

kinleyd wrote: post #42
dajobe wrote: post #11

There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.

Miller looks very useful, and so does q. Thanks to the op and to you for the introductions.

indeed, I hadn't found q. thanks for the intro!

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#56

Post by im3w1l »

jakobegger wrote: post #40

It looks like this tool doesn't support semicolon-separated CSV files, quotes, or numbers using a comma as decimal separator. Too bad. It's a pity that the CSV file format is so fragmented. It's all too common that a CSV file written by tool A can't be read by tool B. Only very few tools do it right. For example, I think the ruby CSV module has a heuristic to automatically detect the "style" of CSV files, which is pr…

Auto-detecting format sounds like a recipe for injection attacks.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#57

Post by burntsushi »

> And initial experiments with Rust are intriguing. Yet with none of them could I obtain the throughput I get in C: see the Performance page.

I'd be curious to see how `xsv` (written in Rust) compares performance wise to Miller. (I can't compile Miller---I commented on the issue tracker.)

> Thus the absence of in-process multiprocessing is only a slight penalty in this particular application domain — parallelism here is more easily achieved by running multiple single-threaded processes, each handling its own input files, either on a single host or split across multiple hosts.

I disagree. CSV files can be indexed[1] very simply, providing random access. This makes it trivial to split a large CSV file into chunks which can be operated on in parallel. This lifts the burden off the user to do the splitting and merging manually, which can be become quite tedious when you want to do frequency analysis on the fields.

This type of indexing is not usually seen because it requires support from the underlying CSV parser to produce correct byte offsets. But you've written your own parser here in Miller, so you should be able to do something similar to `xsv`. Moreover, random access gets you slicing proportional to the slice. i.e., extract the last 10 records in a 4M row CSV file is instanteous. This opens up other doors too, like potentially faster joining and fast random sampling.

P.S. I love the name. :-)

[1] - http://burntsushi.net/rustdoc/csv/index/index.html

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#58

Post by lambda »

dajobe wrote: post #11

There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.

There's also https://github.com/BurntSushi/xsv which is fairly similar in how it works to Miller, though it can also index a CSV file and then future operation will be sped up by utilizing the index.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#59

Post by brudgers »

smegel wrote: post #4

If you work with this sort of data a lot, you might want to look at R.

Your comment brings up an interesting point. Sed, Awk, etc. are single purpose tools and that's one of the reasons their so long lived according to the common wisdom of *Nix. The other side of that wisdom is that once you build a swiss-army-knife, it gets harder to argue against just using a full blown programming language, like R, TCL, Perl, Python, etc.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#60

Post by uxcn »

brudgers wrote: post #59
smegel wrote: post #4

If you work with this sort of data a lot, you might want to look at R.

Your comment brings up an interesting point. Sed, Awk, etc. are single purpose tools and that's one of the reasons their so long lived according to the common wisdom of *Nix. The other side of that wisdom is that once you build a swiss-army-knife, it gets harder to argue against just using a full blown programming language, like R, TCL, Perl, Python, etc.

I use R, and I still use ad-hoc commands from time to time too. I guess mlr and xsv lie somewhere between, which I've personally never really considered.

R is specifically designed for manipulating datasets though, and has a lot more functionality/dependencies. If you're already familiar with the language, and okay with the dependencies, it is vastly more powerful, more succinct, and arguably easier to use. Otherwise, something in between probably still makes sense.

Post reply on HN