11–20 of 57 posts
Sublime Text and VSCode work fine with 1GB files in my experience.
That said, worth mentioning CSV syntax highlighters for text editors. I’ve found rainbow csv quite helpful (https://github.com/mechatroner/sublime_rainbow_csv)
It's definitely not the problem of the resources, but only of the architecture of applications.
(Indeed, the right application is a DBMS, not a spreadsheet.)
No mention of vi or Sqlite? While I'm no vi expert it's a great tool for working with big files when you want to browse around without grep. And Sqlite is similarly ubiquitous and capable of crunching large files.
vi is an editor so it doesn't really solve my requirement of getting spreadsheet-like editing capabilities. SQlite is a good idea, I never thought of that. I will investigate that and add it to the article. Thanks!
I use something like this to fix up column names and import:
sed -i '' -e '1 s/ \/ /_/g; 1 s/[() ]/_/g' $csv_file
sqlite3 $db_file Earlier quoted context omitted.
vi is an editor so it doesn't really solve my requirement of getting spreadsheet-like editing capabilities. SQlite is a good idea, I never thought of that. I will investigate that and add it to the article. Thanks!
SQLite does a great job of importing CSV, and then you're able to use something like "DB Browser for SQLite"[1] to browse the data. I use something like this to fix up column names and import: sed -i '' -e '1 s/ \/ /_/g; 1 s/[() ]/_/g' $csv_file sqlite3 $db_file [1] https://sqlitebrowser.org/
Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
You could get interested in: https://github.com/BurntSushi/xsv
I'm always astounded that there doesn't seem to be a decent general purpose CSV editor/viewer application. Excel is atrocious - it's always dog slow, and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly". Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
Why do you want to do that? Use cat, pipe, grep, awk. Problem solved.
I personally am not a huge fan of awk, I've never built a great mental model around its syntax and it doesn't really solve the problem I was talking about, which is getting a spreadsheet-like editing experience. Thanks for bringing it up, I should definitely add it to the article.
The first tools I reach for when dealing with CSVs of these and larger magnitudes are less, cut, awk, etc. They also tend to be the last tools I end up needing.