https://kothar.net/csview
A fast viewer a friend of mine created to view large CSVs in a GUI - might be useful to someone.
How To Open and Manipulate Large CSV Files On A Mac
41–50 of 57 posts
Re: How To Open and Manipulate Large CSV Files On A Mac
#42I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
The text in a column can be quoted and inside the quotes there can be escaped quotes or commas.
It requires a lot of sed awk skills to merge two columns or delete a column. If at all possible
Re: How To Open and Manipulate Large CSV Files On A Mac
#43There seem to be literally dozens of solutions to do read-only operations but very few to enable comfortable editing of the files in-place in a Unix / command line environment.
Seems like a real gap in the software ecosystem atm:
- fast
- no limit on file size
- spreadsheet style layout
- command line
- easily edit and update individual cell => save
Re: How To Open and Manipulate Large CSV Files On A Mac
#44I'm always astounded that there doesn't seem to be a decent general purpose CSV editor/viewer application. Excel is atrocious - it's always dog slow, and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly". Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
There is csvkit and xsv and I’ve seen a few treat a csv file as a database projects around
Re: How To Open and Manipulate Large CSV Files On A Mac
#45I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
awk, sed, cut, etc... are excellent as long as the format is regular and you only need to process the file once
Re: How To Open and Manipulate Large CSV Files On A Mac
#46I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
awk, sed, cut, etc... are excellent as long as the format is regular and you only need to process the file once
[deleted]
Re: How To Open and Manipulate Large CSV Files On A Mac
#47There seem to be literally dozens of solutions to do read-only operations but very few to enable comfortable editing of the files in-place in a Unix / command line environment. Seems like a real gap in the software ecosystem atm: - fast - no limit on file size - spreadsheet style layout - command line - easily edit and update individual cell => save I've tried VIM's CSV plugins many times and have never been satisfie…
Have you tried VisiData? https://visidata.org.
Re: How To Open and Manipulate Large CSV Files On A Mac
#48Earlier quoted context omitted.
Python + Pandas + Jupyter Notebook/Lab
Python + Jupyter OK, but pandas actually reads everything at once, doesn’t it. 100MB is no problem but bigger files could result in high swapping pression.
I definitely agree that with this amount of data, you should move to a more programmatic way to handle it... pandas or R.
Keep in mind that pandas (and probably also R?) internally uses optimized structures based on numpy. So a 10 GB csv, depending on the content, might end up with a much smnaller memory footprint inside pandas.
If you have 10 GB csv, I think you will be happy working with pandas locally even on a Laptop. If you go to csv files with tens of GB, a cloud vm with corresponding memory might serve you well. If you need to handle big-data-scale csvs (hundreds of GB or even >TB), a scalable parallel solution like Spark will be your thing. Before you scale up however, maybe your task allows to pre-filter the data and reduces the amount by orders of magnitude... often, thinking the problem through reduces the amount of metal one needs to throw at the problem...