Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

johnkerl.org

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#71

Post by pyre »

singingfish wrote: post #65
pyre wrote: post #44

Earlier quoted context omitted.

Well, DBD::CSV is just a driver for DBI, not a command that is "unix pipeline friendly"... So there's that.

Perl is pipeline friendly though ...

DBD::CSV is a library, which you could use to build a command, but doesn't seem to come with one out of the box. q is just a command, but not a library.

One could use DBI + DBD::CSV + Perl to build something similar to q, but that's a batteries not included solution.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#72

Post by acveilleux »

dbro wrote: post #66
acveilleux wrote: post #64

Earlier quoted context omitted.

Not sufficient, but close. It substitutes the quoted character for another one.

Hi- I'm curious to know what your use case is. Can you explain why substitution is not sufficient?

(1) I filter on column content using regex and dealing with a sub character adds complexity.

(2) Many of my columns are free-form text containing commas, carriage returns, new lines, tab, vertical tabs and file separator (0x1c). Occasionally, text is in UCS-2/UTF-16 or uses UTF-8 and foreign characters (a non-trivial quantity of the text I process is in French for example.)

(If you read between the lines here, some columns can contain MLLP-encoded HL7 messages, others contain free-form text and I'm in the medical field.)

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#74

Post by kazinator »

You can name fields in awk so that you don't have to refer to them numerically as $8, $13 or whatever. Example:

   BEGIN { first=1; second=2; third=3 }

   $first > 42 { print $third }
Here, $first means $ applied to the variable first. Since first evaluates to 1, it means $1.

Broken CSV handling that doesn't respect quotes is easily done in awk using comma as the field separator.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#76

Post by dmix »

lambda wrote: post #58
dajobe wrote: post #11

There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.

There's also https://github.com/BurntSushi/xsv which is fairly similar in how it works to Miller, though it can also index a CSV file and then future operation will be sped up by utilizing the index.

xsv is awesome, just used it for a mini-project. Thanks for the heads up.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#77

Post by john_kerl »

john_kerl wrote: post #52
ghshephard wrote: post #20

Earlier quoted context omitted.

Not totally portable. They have built some ctags option dependencies that I'm going to have to futz with in order to get the makefile to run. 12:58 shephard:c shephard$ make all ctags -R . ctags: illegal option -- R usage: ctags [-BFadtuwvx] [-f tagsfile] file ... make: *** [tags] Error 1 [Edit - No "-R" option for ctags in OS X 10.8.5, or in OpenBSD Current - http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/ma…

ctags etc I'll split out. effectively there are make targets to make dev nice, and a smaller set just to make the binary. ctags definitely in the former category. thanks for the feedback!

fixed in master

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#78

Post by john_kerl »

john_kerl wrote: post #45
ghshephard wrote: post #20

Earlier quoted context omitted.

Not totally portable. They have built some ctags option dependencies that I'm going to have to futz with in order to get the makefile to run. 12:58 shephard:c shephard$ make all ctags -R . ctags: illegal option -- R usage: ctags [-BFadtuwvx] [-f tagsfile] file ... make: *** [tags] Error 1 [Edit - No "-R" option for ctags in OS X 10.8.5, or in OpenBSD Current - http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/ma…

yes, there is an issue open. I have . in my path. will post a diff.

fixed in master

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#79

Post by cben »

kazinator wrote: post #74

You can name fields in awk so that you don't have to refer to them numerically as $8, $13 or whatever. Example: BEGIN { first=1; second=2; third=3 } $first > 42 { print $third } Here, $first means $ applied to the variable first. Since first evaluates to 1, it means $1. Broken CSV handling that doesn't respect quotes is easily done in awk using comma as the field separator.

What about non-broken CSV? Is there an easy way to tell awk to skip escaped or quoted delimeters?
Post reply on HN