Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

johnkerl.org

Post reply on HN
Print view View original

1–10 of 81 posts

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#3

Post by john_kerl »

chromano wrote: post #2

I see a lot of value on this tool, because I've been in the same situation as you. I can't thank you enough for this, a tool that I've been wanting to implement for so long.

thanks! i use it regularly. but there are surely many corner cases & bugfixes i haven't encountered yet. i'd be happy to get your feedback.

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#5

Post by ketralnis »

Not exactly the same thing, but I often use a tiny tool I wrote called shellql[1] that lets you just type in SQL like

    cat foo.csv | shellql "select f1, count(*) from tbl where f2='thingiwant' group by f1 order by count(*) desc limit 5"
It has some limitations on the formats of CSV that it can take, namely that string fields have to be quoted, but you can see from the code that there's not much to it if you wanted to change that. It'd also be easy enough to take in multiple CSV files, join them together, etc.

sqlite's pretty badass.

[1] https://gist.github.com/ketralnis/8124b3ead9d94e00dcb2

Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV

#9

Post by acveilleux »

I have my own little toolkit of csv parsing tools that do most of these (using python's csv module.)

I suspect most people who do deal with bulk CSV files all the time do as well. I'd be curious about how gracefully it handles large CSV files.

The lack of quoting support kills it for my use case however. I wrote my own tools, starting with csv_cut because cut(1) didn't do quoting.

Post reply on HN