Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consider Using CSV

jfhr.me

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Consider Using CSV

#101

Post by Macha »

kasajian wrote: post #98
Pinus wrote: post #46

CSV looks deceptively simple. It is far too easy to just write(','.join(whatever)), which sort of works, until it doesn’t, and then someone, sometimes I, has to sort out the resulting mess. PLEASE use a proper CSV library (Python comes with a CSV module in the standard library), or at least implement the entire format according to the RFC from the outset, even if you think you won’t need it!

This is a matter of developer education. The correct way to create and parse CSV files is to use a third-party library. They can get complicated. A field in a CSV can contain commas and quotes. In some cases, a single field can contain a line-feed, and you'll need to ensure the parser you use supports that. This would allow an entire CSV file to be embedded inside the field of a CSV field. At a minimum, a parser must…

Right, but if you're picking CSV, you likely expect to interoperate with a provider that's not yourself. And then there's no way a parser can handle all CSV formats in the wild.

e.g. the example from my comment on the last CSV discussion (https://news.ycombinator.com/item?id=28223719)

What variant is this:

    1,5,Here is a string "" that does stuff,2021-1-1
What is the value of the third column?

Is this a CSV file without quoting? Then it's

    Here is a string "" that does stuff
Or is it a CSV file with double quote escaping? Then it's

    Here is a string " that does stuff

Re: Consider Using CSV

#102

Post by scrollaway »

jointpdf wrote: post #80
scrollaway wrote: post #30

Earlier quoted context omitted.

Parquet has the opposite problem of CSV though. It's so complex to work with, that unless you're specifically in data science, it's both unheard of and unusable. To read a parquet file in Python, you need Apache Arrow and Pandas. And literally the second result for "parquet python libraries" is an article titled "How To Read Parquet Files In Python Without a Distributed Cluster". I remember dealing with Parquet file…

We data scientists are well-known for our exclusive mastery data wrangling arcana, like… df = pandas.read_parquet(‘foo.parquet’) df.to_csv(‘foo.csv’) df.to_json(‘foo.json’) (no sarcasm)—how could it be simpler than that? What problems have you encountered that make it unusable?

Arrow and pandas are massive dependencies.

Re: Consider Using CSV

#103

Post by YmiYugy »

I always thought CSV was just fine, until I had to ingest and export a bunch of CSV in my last project. The big problem is that CSV is not well defined and it's so deceptively simple that many don't bother to adhere to the spec that does exist. Just a few idiosyncrasies I found: Inconsistent character encoding. If you open or save a csv with Excel it will assume a Windows-1252 encoding. Since browsers deal exclusively with UTF-8, this get's really messy. The CSV I got didn't actually use a comma as a delimiter but a semicolon. Everyone seems to have conflicting options about whether strings should have quotes and if so, which ones. The CSV I had to deal with also came with a decimal comma, which screwed up even more stuff. My advice stay away from CSV as an exchange format. Use something that is well defined.

Re: Consider Using CSV

#105

Post by WirelessGigabit »

No. Just no. The amount of times I've had issues with CSVs exported from a non-US locale is insane. They use semi-colon as separator, as for some weird reason they use the comma as the decimal point.

Then there's the issue of encoding, as that is also not the same across locales. Then you get a CSV with the BOM characters up front or some French accents represented as ? because of incorrect encoding parsing / saving.

At least JSON doesn't have any of these things. Standardized strings, and standardized number format.

Re: Consider Using CSV

#106

Post by brundolf »

I worked at a company where we did this for some endpoints and it worked great. Our client app had to request enormous time-series datasets and using CSV cut a significant percentage off of the payload size. I recommend it if you have similar constraints

Re: Consider Using CSV

#107

Post by brundolf »

Pinus wrote: post #46

CSV looks deceptively simple. It is far too easy to just write(','.join(whatever)), which sort of works, until it doesn’t, and then someone, sometimes I, has to sort out the resulting mess. PLEASE use a proper CSV library (Python comes with a CSV module in the standard library), or at least implement the entire format according to the RFC from the outset, even if you think you won’t need it!

Yeah, but this is less of a problem if it's an internal API. You can stick to a stricter subset of the standard, and/or only handle the types of column values that you actually need

Still probably worth using a library, but it isn't a source of problems in my experience

Re: Consider Using CSV

#108

Post by ndsipa_pomu »

mtVessel wrote: post #104
ndsipa_pomu wrote: post #77

Earlier quoted context omitted.

I did not know that. It's on most UK keyboards

What is it called? What's it for?

I had to look it up as I don't know what it's called and it's a negation symbol https://en.wikipedia.org/wiki/List_of_logic_symbols

As to what it's for, I'd say it's great as a delimiter. I've never used it for its intended purpose.

Re: Consider Using CSV

#109

Post by speq »

sitkack wrote: post #75
majkinetor wrote: post #22

Since this is about CSV, this is obligatory tool for larger ones: * https://github.com/antonycourtney/tad

For manipulating CSV from the terminal, check out https://github.com/BurntSushi/xsv

There's a fork with new features: https://github.com/jqnatividad/qsv

Re: Consider Using CSV

#110

Post by majkinetor »

speq wrote: post #109
sitkack wrote: post #75

Earlier quoted context omitted.

For manipulating CSV from the terminal, check out https://github.com/BurntSushi/xsv

There's a fork with new features: https://github.com/jqnatividad/qsv

Unless you really need ultra performance, PowerShell is certainly much better option.
Post reply on HN