CleverCSV: A Drop-In Replacement for Python's CSV Module

Pandas uses the Python csv module under the hood. Most of the code in the read_csv method is used to get the data from the csv file and load it into data frames.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#5

Post by donarb » Mon, Jan 13, 2020, 10:23 PM UTC

MapleWalnut wrote: post #3

r0f1 wrote: post #2

How does this compare to Pandas?

This project appears to have a dependency on pandas. https://github.com/alan-turing-institute/CleverCSV/blob/mast...

That dependency is merely a wrapper to allow CleverCSV to load data directly into a Pandas data frame. It doesn't require pandas just for loading csv data.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#6

Post by etaioinshrdlu » Mon, Jan 13, 2020, 10:43 PM UTC

This seems kind of philosophically similar to FTFY, https://github.com/LuminosoInsight/python-ftfy, except for CSV files instead of botched text strings.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#7

Post by scrollaway » Mon, Jan 13, 2020, 11:21 PM UTC

etaioinshrdlu wrote: post #6

This seems kind of philosophically similar to FTFY, https://github.com/LuminosoInsight/python-ftfy , except for CSV files instead of botched text strings.

I knew of ftfy already but I would love some more examples of these kinds of data cleanup libraries; it's always super useful.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#8

Post by urschrei » Mon, Jan 13, 2020, 11:52 PM UTC

donarb wrote: post #4

r0f1 wrote: post #2

How does this compare to Pandas?

Pandas uses the Python csv module under the hood. Most of the code in the read_csv method is used to get the data from the csv file and load it into data frames.

Are you sure? Wes McKinney rewrote the Pandas CSV parser way back in late 2012 because the batteries-included version was slow and memory-hungry. Did Python upstream his version later on, or replace it with something faster (I'd be surprised if they did either, but I'd be curious to know)

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#9

Post by gregmac » Tue, Jan 14, 2020, 12:16 AM UTC

> standardize Convert a CSV file to one that conforms to RFC-4180.

Very glad to see this here, hopefully it'll help with increasing adoption of the RFC.

I find one of the biggest pains of working with CSV is that all too often, someone gets Excel involved, and then Excel absolutely butchers the file. Off the top of my head: It strips leading 0's (even if quoted), converts big numbers to scientific notation, sometimes interprets non-date values as dates, and when saving, keeps the mangled values, doesn't preserve quotes, and uses regional settings -- to the point in some locales, it becomes a "semi-colon separated file"! Even just opening and then saving a .csv (without editing) can totally mangle it to the point it's unusable. Sometimes Excel can't even open a file saved by Excel without mangling it even more.

[1] https://tools.ietf.org/html/rfc4180

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#10

Post by emmanueloga_ » Tue, Jan 14, 2020, 12:59 AM UTC

I learnt a while ago about a way to provide a schema for CSV files. If you publish CSV data, may be useful to provide a schema for it [1].

Sounds like CleverCSV could determine and generate a schema automatically given a CSV file.

1: https://github.com/theodi/csvlint.rb#json-table-schema-suppo...