Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

miller.readthedocs.io

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#11

Post by bee_rider »

vcdimension wrote: post #4

This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).

The version in the Ubuntu repos is 5.10. I wonder if they'll "upgrade" to 6.0. Downthread people are complaining about the compilation time for the C version, which seems like a much less significant issue when the cost of compilation is amortized over many users. But if Go is a much more productive language, I guess a feature delta will accumulate...

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#12

Post by marginalia_nu »

tinix wrote: post #7
svnpenn wrote: post #5

Earlier quoted context omitted.

Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.

That's a fascinating perspective... This mentality is why modern software seems slower... Because it is. So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance. It's a shame, really...

It's basically Parkinson's law. If you give someone 10x the performance, they'll find a way of doing the same things they did before using 10x more resources.

It's why you have user interfaces today that are often as slow as, or slower than the equivalent Windows 3.11 interfaces. In 1993 things took time because the processor was an actual potato and the data fetched off a floppy disk, today things are slow because the code being run is 1-10k times slower and messages need to be continuously sent to a datacenter half way around the world to inform an analytics server about your every input.

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#13

Post by lettergram »

My team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command:

https://github.com/capitalone/DataProfiler

Effectively loads anything into a dataframe. After that you can profile the data, merge, save, load, and take differences between profiles before generating a report.

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#14

Post by vjust »

lettergram wrote: post #13

My team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command: https://github.com/capitalone/DataProfiler Effectively loads anything into a dataframe. After that you can profile the data, merge, save, load, and take differences between profiles before generating a report.

Cool! I will give this a spin.

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#16

Post by dima55 »

jasonpeacock wrote: post #8

It's interesting watching these types of tools get re-invented periodically: https://github.com/benbernard/RecordStream It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.

Lots of them out there. Here's one that repurposes the extends the standard unix tools to know about comments and column labels: https://github.com/dkogan/vnlog/

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#17

Post by loudmax »

jasonpeacock wrote: post #8

It's interesting watching these types of tools get re-invented periodically: https://github.com/benbernard/RecordStream It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.

Pretty sure JSON isn't part of POSIX.

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#18

Post by dang »

CodeIsTheEnd wrote: post #3

Previously discussed (66 comments) just a few months ago: https://news.ycombinator.com/item?id=28298729

Thanks! Macroexpanded:

Miller CLI – Like Awk, sed, cut, join, and sort for CSV, TSV and JSON - https://news.ycombinator.com/item?id=28298729 - Aug 2021 (66 comments)

Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions - https://news.ycombinator.com/item?id=13751389 - Feb 2017 (20 comments)

Miller = sed, awk, cut, join, sort for CSV and tabular JSON - https://news.ycombinator.com/item?id=11674304 - May 2016 (1 comment)

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV - https://news.ycombinator.com/item?id=10073040 - Aug 2015 (2 comments)

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV - https://news.ycombinator.com/item?id=10066742 - Aug 2015 (75 comments)

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#19

Post by Zababa »

vcdimension wrote: post #4

This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).

I hope that this is temporary to make working on the project easier, Go can get pretty close to C in terms of speed and it's usually easier to jump into a Go project. But 10 times slower seems way too much.

Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

#20

Post by pdimitar »

dima55 wrote: post #16
jasonpeacock wrote: post #8

It's interesting watching these types of tools get re-invented periodically: https://github.com/benbernard/RecordStream It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.

Lots of them out there. Here's one that repurposes the extends the standard unix tools to know about comments and column labels: https://github.com/dkogan/vnlog/

If have any more links, please post them! I am a fan of collecting such tools.
Post reply on HN