I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
How To Open and Manipulate Large CSV Files On A Mac
31–40 of 57 posts
Re: How To Open and Manipulate Large CSV Files On A Mac
#32Memory mapped file.. Intermediate binary representation..
[deleted]
Re: How To Open and Manipulate Large CSV Files On A Mac
#33Python + Pandas
Or R + Tidyverse will do the job nicely too.
Re: How To Open and Manipulate Large CSV Files On A Mac
#34Re: How To Open and Manipulate Large CSV Files On A Mac
#35I'm always astounded that there doesn't seem to be a decent general purpose CSV editor/viewer application. Excel is atrocious - it's always dog slow, and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly". Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
http://visidata.org/ Is pretty fast if you’re comfortable in a terminal (not cli) app.
Re: How To Open and Manipulate Large CSV Files On A Mac
#36I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
This is the best answer if the csv file is more or less standard.
Re: How To Open and Manipulate Large CSV Files On A Mac
#37Earlier quoted context omitted.
For the quoted spreadsheet-like operations of filtering and rearranging, awk is perfect as a deferred editor. That the kludgy first step of your chosen solution ("First, you must create a CSV file contain only the first 10-20 lines of your large CSV file") isn't just `head very_large_nov_2019.csv > very_large_nov_2019_abridged.csv` seems to further indicate an unfamiliarity with the large set of built-in, battle-test…
How well do those tools work with arbitrary CSV files, e.g. containing line breaks or quotes in field data? I wasn't aware that they can actually parse CSV and instead you have to assume things about the content that may not end up being true.
Every data processing task has to make assumptions about the well-formedness of its input. "Arbitrary CSV" is basically undefined; whether deviations are best dealt with by parsing, preprocessing, or different tools altogether depends on the source.
Re: How To Open and Manipulate Large CSV Files On A Mac
#38I use XSV: A fast CSV command line toolkit written in Rust. https://formulae.brew.sh/formula/xsv https://github.com/BurntSushi/xsv
You can use EBay tools as well, such as https://github.com/eBay/tsv-utils after importing a CSV file as TSV.
Re: How To Open and Manipulate Large CSV Files On A Mac
#39Re: How To Open and Manipulate Large CSV Files On A Mac
#40I'm always astounded that there doesn't seem to be a decent general purpose CSV editor/viewer application. Excel is atrocious - it's always dog slow, and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly". Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
> and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly".
You should be using the Data -> From Text importer, not just double clicking. Also use the latest 64 bit version of Excel.