Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CleverCSV: A Drop-In Replacement for Python's CSV Module

github.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#31

Post by jimnotgym »

turbinerneiter wrote: post #25
jimnotgym wrote: post #23

Earlier quoted context omitted.

For ecommerce sites it is so common to want to go from Excel to UTF8 csv that a lot of people use Libre/Open Office to open and save the Excel file. Then you can do safe things like double-quoting all fields, and choosing your encoding. Perhaps more systems should have Excel upload rather than insisting on UTF-8 csvs that the most common spreadsheet software has never been able to produce?

Maybe the most common spreadsheet software should get it's act together and support this simple standard?

But wouldn't it help the non technical user more if Magento, Shopify etc supported the much more widely used standard of xlsx? Then people don't have to find out what UTF-8 even is. Library support for reading Excel docs is pretty good now, isn't it?

I wish Excel did support csv better, but we are in the .001% of Excel users who even know that it doesn't do a good job.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#32

Post by jimnotgym »

Twirrim wrote: post #13
gregmac wrote: post #9

> standardize Convert a CSV file to one that conforms to RFC-4180. Very glad to see this here, hopefully it'll help with increasing adoption of the RFC. I find one of the biggest pains of working with CSV is that all too often, someone gets Excel involved, and then Excel absolutely butchers the file. Off the top of my head: It strips leading 0's (even if quoted), converts big numbers to scientific notation, sometimes…

At a previous company, someone decided to experiment with putting a comma in a text field submitted to an internal service that for some godforsaken reason spat out CSV files that a lot of other teams consumed. I forget the exact entry in the field but it was something like "I wonder, how does this handle commas". Turns out it didn't handle them well. It didn't escape them or anything. Out spits this line in the csv…

This is actually frustratingly common with off-the-shelf ERP 'integrations'. Things like courier labelling software come to mind as culprits. And you are stuck between trying to get UPS to fix a system that they give away free, or training sales operators to write addresses properly.

I love the ones that present a REST api but still can't handle commas, because they have wrapped some ancient importer in a front end that simply writes the request data to a csv! I had to write one once, but I double quoted all fields and had to translate UTF-8 back to some other standard the old ERP used! It was junk, but I believe is still chugging away years later...

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#33

Post by gjjvdburg »

fake-name wrote: post #27

The only issue I've ever had with python's CSV module is that it's slow . Is this any faster?

Hi, I'm the author of CleverCSV. Although the package was initially written to mainly improve the accuracy of automated dialect detection, we ended up changing some aspects of the C parser as well (most notably increasing the look-ahead to two characters). This has made the parsing somewhat faster than the builtin csv package (around 10-20%).

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#34

Post by bronzecarnage »

Maxious wrote: post #12
Waterluvian wrote: post #11

Earlier quoted context omitted.

That is hilarious, amazing, sad, and begging for a blog post of someone walking through the nightmare you described.

One in five genetics papers contains errors thanks to Microsoft Excel https://genomebiology.biomedcentral.com/articles/10.1186/s13...

Nature has the highest percentage of affected papers. Huh.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#35

Post by tialaramex »

jimnotgym wrote: post #31
turbinerneiter wrote: post #25

Earlier quoted context omitted.

Maybe the most common spreadsheet software should get it's act together and support this simple standard?

But wouldn't it help the non technical user more if Magento, Shopify etc supported the much more widely used standard of xlsx? Then people don't have to find out what UTF-8 even is. Library support for reading Excel docs is pretty good now, isn't it? I wish Excel did support csv better, but we are in the .001% of Excel users who even know that it doesn't do a good job.

XLSX isn't actually a standard in a meaningful sense. So while support for whatever parts are standard is "pretty good" how does that help your users when some of their sheets don't work correctly? Who wants to explain what's wrong and that nobody can fix it but er... maybe better luck next time?

XLSX (Office Open XML) is full of safety valves that let Microsoft's existing products continue doing whatever poorly documented or undocumented stuff they were doing previously. Microsoft didn't want to have to go back to features which worked in Office already and either rip them out or re-implement them, and it didn't have documentation for those features that anybody else would be able to implement‡. So Office Open XML just says in those cases well here's a blob of data and good luck unless you're Microsoft Office.

This is tolerable for exporting to Excel. I can emit compliant Office Open XML that gets my numbers into XL reliably. So that's nice.

But when importing from Excel you're fighting that impedance mismatch. Rather than explain to users "Something about your document is incompatible and I swear it's Microsoft's fault" it's just better to say "Use CSV".

‡ e.g. suppose there's a line in Excel which defines a function FOO() by calling into some particular Windows DLL. Well that's not a useful thing to standardise. So do you call the relevant MS department and ask them to paste all their documentation for that DLL into your "spreadsheet" standard? No. You write "Implementation defined" and it becomes a black box.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#36

Post by joshvm »

scrollaway wrote: post #7
etaioinshrdlu wrote: post #6

This seems kind of philosophically similar to FTFY, https://github.com/LuminosoInsight/python-ftfy , except for CSV files instead of botched text strings.

I knew of ftfy already but I would love some more examples of these kinds of data cleanup libraries; it's always super useful.

This software (Trifacta Wrangler) was demoed at NeurIPS this year and looked very impressive. https://www.trifacta.com/products/wrangler-editions

There is also the non-maintaned version that the company spun out from:

http://vis.stanford.edu/wrangler/

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#37

Post by modo_mario »

jimnotgym wrote: post #31
turbinerneiter wrote: post #25

Earlier quoted context omitted.

Maybe the most common spreadsheet software should get it's act together and support this simple standard?

But wouldn't it help the non technical user more if Magento, Shopify etc supported the much more widely used standard of xlsx? Then people don't have to find out what UTF-8 even is. Library support for reading Excel docs is pretty good now, isn't it? I wish Excel did support csv better, but we are in the .001% of Excel users who even know that it doesn't do a good job.

XLSX is not remotely the same as CSV. CSV is way more widely used than you think under the hood of stuff and is a muuuuuch more slimmed down format. It's litterally just values separated by a delimiter. XLSX also isn't a standard. It'll keep changing.

It's one of those reasons why Microsoft tried to shoehorn their own trash Open document format that would fit them and let them keep backwards compatibility with stoneage versions of their office suite. https://www.computerweekly.com/news/2240225262/Microsoft-att...

It's like saying everyone should have adapted to internet explorer rather than following proper web standards. Even if XSLX was some kind of standard it's a bad idea because at the end of the day MS shouldn't be trusted with those.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#38

Post by closed »

th0ma5 wrote: post #22

The problem mostly with CSV beyond the lack of standards, character encodings, and lack of compression, inability to efficiently write or read, and problems handling multiline freeform text, is that ultimately the structure is inband with the data, which is what makes all problems probable.

> The problem mostly with CSV... is that ultimately the structure is inband with the data

I wonder if for 90% of users this seems like a critical UI feature!

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#39

Post by mhd »

rusk wrote: post #29
mhd wrote: post #28

Earlier quoted context omitted.

Excel is mostly responsible for wrecking any attempt of "standard" CSV, not that there ever was a chance. If I remember correctly, non-Anglo CSVs using a semi-colon as a delimiter is fully to blame on it.

Its not just non-Anglo .. semicolon is often used because data often contains commas, and it’s just easier to swop your delimiter than encapsulate your fields properly ... even then it’s just a simple matter to use the alternative delimeter, or convert by a simple search and replace (once youve added encapuslation to the relevant fields of course). Though CSV directly implies commas I find it helpful to consider it a…

Not saying it's not useful, just that this arose from them not using a hard-coded comma, but referencing some default separator, that happens to be the semi-colon in some locales. I might be wrong and this could have an older precedent.

Now, whether it would've been better to stick with the comma and just properly quote all the values for locales where the comma is a decimal separator and thus often used in spreadsheet columns is a moot point, the damage has been done.

Personally I just call it "character separated values" anyway. I quite like tab-separated values. Makes it okay to read quite often and I think it's the default output of Postgres' COPY command.

Re: CleverCSV: A Drop-In Replacement for Python's CSV Module

#40

Post by manor »

TSV --> Problem Solved! Has anyone ever stumbled upon a cell entry that needed to contain an actual tab character? No!!! The tab character's entire raison-d'etre is to separate fields, whereas the comma separates words. Technically the "C" in CSV denotes "Character" but with TSV the intent is 100% clear. The MIME type is explicit: "Note that fields that contain tabs are not allowable in this encoding." This is a vanishingly small price to pay for a parser a simple as split("\t")...

[1] http://www.iana.org/assignments/media-types/text/tab-separat...

Post reply on HN