Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

171–180 of 711 posts

Re: A love letter to the CSV format

#171

Post by nomel »

simonw wrote: post #154
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

Right - the JSON-newline equivalent of CSV can look like this: ["id", "species", "nickname"] [1, "Chicken", "Chunky cheesecakes"] [2, "Dog", "Wagging wonders"] [3, "Bunny", "Hopping heroes"] [4, "Bat", "Soaring shadows"]

Remove the [] characters and you've invented CSV with excel style quoting.

Re: A love letter to the CSV format

#172

Post by dwattttt »

hajile wrote: post #163
rcbdev wrote: post #153

Earlier quoted context omitted.

Same for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.

JSON has a clearly-defined standards: ISO/IEC 21778:2017, IETF RFC 7159, and ECMA-404. Additionally, Crockford has had a spec available on json.org since it's creation in 2001. Do you have any examples of Python, Java, or any of the other Tiobe top 40 languages breaking the JSON spec in their standard library? In contrast, for the few of those that have CSV libraries, how many of those libraries will simply fail to p…

You need more than a standard; that standard has to be complete and unambiguous. What you're looking for is https://github.com/nst/JSONTestSuite

EDIT: The readme's results are from 2016, but there's more recent results (last updated 5 years ago). Of the 54 parsers /versions tested, 7 gave always the expected result per the spec (disregarding cases where the spec does not define a result).

Re: A love letter to the CSV format

#173

Post by LegionMammal978 »

corysama wrote: post #141
kevmo314 wrote: post #33

Earlier quoted context omitted.

What do you mean? I just push the Record Separator key on my keyboard. /s in case :)

The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not…

> The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today."

There's also the argument of "Now you have two byte values that cannot be allowed to appear in a record under any circumstances. (E.g., incoming data from uncontrolled sources MUST be sanitized to reject or replace those bytes.)" Unless you add an escaping mechanism, in which case the argument shifts to "Why switch from CSV/TSV if the alternative still needs an escaping mechanism?"

Re: A love letter to the CSV format

#174

Post by orthoxerox »

andrepd wrote: post #24
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

But what if one of your columns contains arbitrary binary data?

Re: A love letter to the CSV format

#175

Post by saint_yossarian »

williamcotton wrote: post #81

Essential CSV shell tools: csvtk: https://bioinf.shenwei.me/csvtk/ gawk: https://www.gnu.org/software/gawk/manual/html_node/Comma-Sep... awk: https://github.com/onetrueawk/awk?tab=readme-ov-file#csv

Also VisiData is an excellent TUI spreadsheet.

Re: A love letter to the CSV format

#177

Post by Yomguithereal »

williamcotton wrote: post #81

Essential CSV shell tools: csvtk: https://bioinf.shenwei.me/csvtk/ gawk: https://www.gnu.org/software/gawk/manual/html_node/Comma-Sep... awk: https://github.com/onetrueawk/awk?tab=readme-ov-file#csv

I would add xan to this list: https://github.com/medialab/xan

But of course, I am partial ;)

Re: A love letter to the CSV format

#178

Post by nomel »

Someone1234 wrote: post #107
fragmede wrote: post #54

Earlier quoted context omitted.

I want to push Sqlite as a data interchange format! it has the benefit of being well defined, and can store binary data, like images for product pictures inside the database. not a good idea if you're trying to serve users behind a web app, but as interchange, better than a zip file with filenames that have to be "relinked".

For context: I have a LOT of experience of interchange formats, like "full time job, every day, all day, hundreds of formats, for 20-years" experience. Based on that experience I have come to one key, but maybe, counter-intuitive truth about interchange formats: - Too much freedom is bad . Why? Generating interchange data is cheaper than consuming it, because the creator only needs to consider the stuff they want to…

Oh, this is interesting. Are you tying different systems together? If so, do you use some preferred intermediate format? Do you have a giant library of * -> intermediate -> * converters that you sprinkle between everything? Or maybe the intermediate format is in memory?

What about Parquet and the like?

Re: A love letter to the CSV format

#179

Post by masfuerte »

nly wrote: post #113
masfuerte wrote: post #95

The fact that you can parse CSV in reverse is quite cool, but you can't necessarily use it for crash recovery (as suggested) because you can't be sure that the last thing written was a complete record.

Last field rather than last record. The first row will give you column count.

The column count doesn't help because you don't know where the last record starts because you don't know whether you're in a string or not. Unless you scan the entire file from the beginning, which defeats the object.

Re: A love letter to the CSV format

#180

Post by gopher_space »

int_19h wrote: post #134
benwilber0 wrote: post #114

Earlier quoted context omitted.

JSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and q…

The problem with CSV is that there's no clear standard, so even if you do reach for a library to parse it, that doesn't ensure compatibility.

There is a clear standard and it's usually written on an old Word '97 doc in a local file server. Using CSV means that you are the compatibility layer, and this is useful if you need firm control or understanding of your data.

If that sounds like a lot of edge-case work keep in mind that people have been doing this for more than half a century. Lots of examples and notes you can steal.

Post reply on HN