csvkit: Command-line tools for working with CSV

#1

Post by mxgr » Fri, Jan 20, 2023, 9:35 AM UTC

csvkit: Command-line tools for working with CSV
csvkit.readthedocs.io

Re: csvkit: Command-line tools for working with CSV

#2

Post by mattewong » Fri, Jan 20, 2023, 8:31 PM UTC

I wanted so much to use csvkit and all the features it had, but its horrendous performance made it unscalable and therefore the more I used it, the more technical debt I accumulated.

This was one of the reasons I wrote zsv (https://github.com/liquidaty/zsv). Maybe csvkit could incorporate the zsv engine and we could get the best of both worlds?

Examples (using majestic million csv):

---

csvcut -c 1,3 = 5.3 seconds

zsv select -n -- 1 3 = 0.19 seconds

28x faster

---

csvsql --query "select count(*) from file" file.csv = 148 seconds

zsv sql "select count(*) from data" file.csv = 0.68 seconds

216x faster

---

Re: csvkit: Command-line tools for working with CSV

#3

Post by hermitcrab » Fri, Jan 20, 2023, 8:54 PM UTC

It is interesting how much different tools vary in their performance for the same task. For example R with data.table is much faster than base R. And Excel Power Query performance is, well, see for yourself:

https://www.easydatatransform.com/data_wrangling_etl_tools.h...