This thing is convenient, but I don't understand why it needs a DSL? Why do I need to use `mrl --csv sort ...`, instead of `mrl --csv ... | sort`?
Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
31–40 of 81 posts
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#32Can anyone suggest something similar for Windows?
PowerShell can handle CSV files natively. Here are the equivalent commands from the article: Import-Csv example.csv | Select-Object year, price | Export-Csv -NoTypeInformation example-cut.csv Import-Csv example.csv | Sort-Object year | Export-Csv -NoTypeInformation example-sorted.csv Import-Csv example.csv | Where-Object year -eq 1999 | Export-Csv -NoTypeInformation example-filtered.csv PowerShell can also deal with…
PowerShell has solved this problem incredibly well. It often feels more Unix in its philosophy, than some of the newer *nix tools, I've encountered.
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#33There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.
This is great. Wonder why they didn't go with "cq" or something, though.
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#34This thing is convenient, but I don't understand why it needs a DSL? Why do I need to use `mrl --csv sort ...`, instead of `mrl --csv ... | sort`?
Because otherwise it will sort the header line. It looks like the reason for the --csv is because csv isn't the default format.
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#35I am surprised no ones looking at Apache Drill. https://drill.apache.org
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#36I love csvkit[0] for this purpose [0] https://github.com/onyxfish/csvkit
simple csvkit tutorial:
"
- Convert Excel to CSV: in2csv data.xls > data.csv
- Convert JSON to CSV: in2csv data.json > data.csv
- Print column names: csvcut -n data.csv
- Select a subset of columns: csvcut -c column_a,column_c data.csv > new.csv
- Reorder columns: csvcut -c column_c,column_a data.csv > new.csv
- Find rows with matching ells: csvgrep -c phone_number -r 555-555-\d{4}" data.csv > matching.csv
- Convert to JSON: csvjson data.csv > data.json
- Generate summary statistics: csvstat data.csv
- Query with SQL: csvsql --query "select name from data where age > 30" data.csv > old_folks.csv
- Import into PostgreSQL: csvsql --db postgresql:///database --insert data.csv
- Extract data from PostgreSQL:: sql2csv --db postgresql:///database --query "select * from data" > extract.csv
"
see more: http://csvkit.readthedocs.org/en/0.9.1/#
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#37I love csvkit[0] for this purpose [0] https://github.com/onyxfish/csvkit
csvkit is excellent. I like `csvlook` for display csv files in the terminal.
Can anyone recommend an easy way to convert csv to xls on Linux? I know that libreoffice and gnumeric can do it in headless mode, but it seems terrible overkill.
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#38There is also the (impossible to google) https://github.com/harelba/q that lets you SQL over CSV and is unix pipeline friendly.
does it bring any advantages over https://metacpan.org/pod/DBD::CSV? (not being perl is not an advantage).
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#39I have my own little toolkit of csv parsing tools that do most of these (using python's csv module.) I suspect most people who do deal with bulk CSV files all the time do as well. I'd be curious about how gracefully it handles large CSV files. The lack of quoting support kills it for my use case however. I wrote my own tools, starting with csv_cut because cut(1) didn't do quoting.
Regarding quoting support, check out https://github.com/dbro/csvquote
Re: Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV
#40It looks like this tool doesn't support semicolon-separated CSV files, quotes, or numbers using a comma as decimal separator. Too bad.
It's a pity that the CSV file format is so fragmented. It's all too common that a CSV file written by tool A can't be read by tool B.
Only very few tools do it right. For example, I think the ruby CSV module has a heuristic to automatically detect the "style" of CSV files, which is pretty neat.