Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How To Open and Manipulate Large CSV Files On A Mac

alecdibble.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: How To Open and Manipulate Large CSV Files On A Mac

#21

Post by banku_brougham »

tad is absolutely great for this, i tested it for the exact use case mentioned. I had to file a ticket and wait for him to add an export to csv function. tad is built over sqlite and can filter/sort/pivot/aggregate and export the result, which is all my business team partners ever need. can handle GB files, i didnt test with TB size.

https://www.tadviewer.com/, its a desktop app

edit: one note after seeing other comments based on cli (xsv, sed, awk, etc) — the OP’s use case is something that marketing/pm/business stakeholders can use. my favorite tools are cli-based, however this does not fly with business teams so forget that option.

Re: How To Open and Manipulate Large CSV Files On A Mac

#22

Post by codesnik »

mamcx wrote: post #14

You could get interested in: https://github.com/BurntSushi/xsv

nice. long time ago I had much success importing and filtering multiple gigabytes of CSV data into elasticsearch using CSVfix (if I'm not mistaken) + jq (converting it to json-lines in between, using jq as well) xsv seems to cover some areas I've used jq.

Re: How To Open and Manipulate Large CSV Files On A Mac

#23

Post by ygra »

bangonkeyboard wrote: post #20
alecdibble wrote: post #3

Earlier quoted context omitted.

I personally am not a huge fan of awk, I've never built a great mental model around its syntax and it doesn't really solve the problem I was talking about, which is getting a spreadsheet-like editing experience. Thanks for bringing it up, I should definitely add it to the article.

For the quoted spreadsheet-like operations of filtering and rearranging, awk is perfect as a deferred editor. That the kludgy first step of your chosen solution ("First, you must create a CSV file contain only the first 10-20 lines of your large CSV file") isn't just `head very_large_nov_2019.csv > very_large_nov_2019_abridged.csv` seems to further indicate an unfamiliarity with the large set of built-in, battle-test…

How well do those tools work with arbitrary CSV files, e.g. containing line breaks or quotes in field data? I wasn't aware that they can actually parse CSV and instead you have to assume things about the content that may not end up being true.

Re: How To Open and Manipulate Large CSV Files On A Mac

#27

Post by xrftghvb »

bArray wrote: post #26

@alecdibble Could you try LibreOffice's Calc [1]? It's my daily driver in Linux, it mostly works well as an excel replacement but I'm interested to see how well it does on a Mac. [1] https://www.libreoffice.org/download/download/

Maximum number of Rows per worksheet = 1 048 576 (2^20).

Re: How To Open and Manipulate Large CSV Files On A Mac

#29

Post by bArray »

xrftghvb wrote: post #27
bArray wrote: post #26

@alecdibble Could you try LibreOffice's Calc [1]? It's my daily driver in Linux, it mostly works well as an excel replacement but I'm interested to see how well it does on a Mac. [1] https://www.libreoffice.org/download/download/

Maximum number of Rows per worksheet = 1 048 576 (2^20).

Apparently you should be able to go way above this [1].

[1] https://stackoverflow.com/questions/41213869/where-to-find-t...

Post reply on HN