Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
miller.readthedocs.io
Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
1–10 of 34 posts
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#2I'll give this the second best compliment: I can't believe amongst SED, AWK and so on, this doesn't already exist.
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#3Previously discussed (66 comments) just a few months ago:
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#4This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#5This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).
Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#6Ive wrote my own tool for doing this, just with cli style commands. However I found that at a certain level of complexity, python is generally easier. With VScode you can run the script in the terminal and view the output file change in a tab immediately.
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#7This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).
Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
That's a fascinating perspective...
This mentality is why modern software seems slower... Because it is.
So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance.
It's a shame, really...
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#8It's interesting watching these types of tools get re-invented periodically:
https://github.com/benbernard/RecordStream
It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#9Earlier quoted context omitted.
Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
That's a fascinating perspective... This mentality is why modern software seems slower... Because it is. So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance. It's a shame, really...
Well, compared to Go, Rust, PHP, or many other modern tools, C is garbage in terms of build experience. Speed is important, but its not the only factor.
Re: Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
#10This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).
Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
You'd rather optimize for compile time, which you do (maybe) 1x, over runtime, which you easily do >100x?
You're focusing on the developer, not the user, but this is a user tool :(