Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consider Using CSV

jfhr.me

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Consider Using CSV

#41

Post by xwowsersx »

akhmatova wrote: post #21
xwowsersx wrote: post #12

I mean point well taken, but, as they acknowledged in the post themselves, CSV isn't suitable when you have a nested structure. And you almost always have/need a nested structure, no?

CSV isn't suitable when you have a nested structure. As the post acknowledges right about where you stopped skimming. And you almost always have/need a nested structure, no? No.

> as they acknowledged in the post themselves

As I noted in my own comment. Ironic to accuse me of skimming the original post when you couldn't even read my two sentences.

Re: Consider Using CSV

#42

Post by CharlesW »

majkinetor wrote: post #19
CharlesW wrote: post #14

Earlier quoted context omitted.

As Wikipedia puts it, "CSV is widely used to refer to a large family of formats that differ in many ways". If there's a canonical standard, it appears to be RFC4180: https://www.rfc-editor.org/rfc/rfc4180

It appears, but its not. I have not found single program so far that conforms only to this RFC and nothing else. From the RFC itself: Status of This Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

> I have not found single program so far that conforms only to this RFC and nothing else.

Wouldn't that be impossible, given that parsers have to accept all kind of bizarro CSV flavors? Maybe more importantly, do you know of a single program or single CSV library that doesn't support reading or writing CSV as defined by the RFC?

Re: Consider Using CSV

#43

Post by gregmac »

ndsipa_pomu wrote: post #3

As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.

There is a spec (RFC 4180 [1]) but it's definitely not widely followed. Worse, for a lot of data there's no problems for potentially years, until your numbers get too big or the first time a quote or comma gets in the data.

In my experience one of the biggest barriers I run into -- and the primary reason I hate using CSV -- is Microsoft Excel. It misinterprets numbers as dates, it convers big numeric identifiers to exponents, and more. Even merely opening a RFC4180-compliant file and saving it changes the data, and even Excel itself will often have a different misinterpretation of the de file.

If humans never used Excel for CSV, it would be a viable format. At the same time in most cases where humans aren't in the loop (machine-to-machine communications), there's better formats. You could spec "RFC4180 CSV" and hope no developer just sees the "CSV" and assumes they understand. Or specify something like a JSON streaming format and avoid a whole lot of headache.

[1] https://www.ietf.org/rfc/rfc4180.txt

Re: Consider Using CSV

#44

Post by margarina72 »

you may also simply add a format specification and return either csv or json depending on the need or the context. Most language would have what it needs to return either without much trouble.

Re: Consider Using CSV

#45

Post by kevinmgranger »

elcritch wrote: post #31
fatneckbeardz wrote: post #24

Earlier quoted context omitted.

but which delimiter. if you choose pipe ok, now you have to make sure nobody typed a pipe into the input field or spreadsheet, and you cannot store unix commands if you choose tab, ok, now people will get confused when they try to edit the text file to replace tabs with spaces, and now you have trouble putting code snippets into data fields because they have tabs. this is the problem and it's why xml/json exist. in m…

Well the obvious solution would be ASCII 0x1D (Group Separator)! Accept, no one actually uses those ASCII characters. Kind of bums me out that UNIX basically skipped out on them.

It's not a separator character, but at least vim and emacs acknowledge the page feed character. A pittance, I suppose.

Re: Consider Using CSV

#46

Post by Pinus »

CSV looks deceptively simple. It is far too easy to just write(','.join(whatever)), which sort of works, until it doesn’t, and then someone, sometimes I, has to sort out the resulting mess. PLEASE use a proper CSV library (Python comes with a CSV module in the standard library), or at least implement the entire format according to the RFC from the outset, even if you think you won’t need it!

Re: Consider Using CSV

#47

Post by akhmatova »

xwowsersx wrote: post #41
akhmatova wrote: post #21

Earlier quoted context omitted.

CSV isn't suitable when you have a nested structure. As the post acknowledges right about where you stopped skimming. And you almost always have/need a nested structure, no? No.

> as they acknowledged in the post themselves As I noted in my own comment. Ironic to accuse me of skimming the original post when you couldn't even read my two sentences.

My very bad - please have my sincerest apology.

Re: Consider Using CSV

#48

Post by xwowsersx »

akhmatova wrote: post #47
xwowsersx wrote: post #41

Earlier quoted context omitted.

> as they acknowledged in the post themselves As I noted in my own comment. Ironic to accuse me of skimming the original post when you couldn't even read my two sentences.

My very bad - please have my sincerest apology.

Apology accepted, no worries

Re: Consider Using CSV

#49

Post by CharlesW »

scrollaway wrote: post #30
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

Parquet has the opposite problem of CSV though. It's so complex to work with, that unless you're specifically in data science, it's both unheard of and unusable. To read a parquet file in Python, you need Apache Arrow and Pandas. And literally the second result for "parquet python libraries" is an article titled "How To Read Parquet Files In Python Without a Distributed Cluster". I remember dealing with Parquet file…

> It's so complex to work with, that unless you're specifically in data science, it's both unheard of and unusable.

FWIW, in my experience at a "data analytics platform" company, it's reasonably popular for data-heavy workflows since Parquet is well-defined, and file sizes (especially as the amount of data grows) are a fraction of their CSV equivalents.

> Is it a limitation of the format itself?

I don't think so. In other languages, you can generally read/write Parquet files without a ton of dependencies (e.g. https://github.com/xitongsys/parquet-go).

Re: Consider Using CSV

#50

Post by panzerboiler »

I usually prefer a binary encoding. More efficient on the wire, easier to parse and generate, and with no ambiguity. We have 2 control codes given to us by the teletype era that have the perfect meaning for this kind of data:

    0x1E Record Separator
    0x1F Unit Separator
Post reply on HN