Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

211–220 of 711 posts

Re: A love letter to the CSV format

#211

Post by nextts »

taeric wrote: post #69
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

Reminds me of a fatal flaw of yaml. Turns out truncating a yaml file doesn't make it invalid. Which can lead to some rather non-obvious failures.

What is the failure mode where a yaml file gets truncated? They are normally config files in Git. Or uploaded to S3 or Kubernetes etc.

CSV has the same failure mode. As does HTML. (But not XML)

Re: A love letter to the CSV format

#212

Post by nly »

masfuerte wrote: post #179
nly wrote: post #113

Earlier quoted context omitted.

Last field rather than last record. The first row will give you column count.

The column count doesn't help because you don't know where the last record starts because you don't know whether you're in a string or not. Unless you scan the entire file from the beginning, which defeats the object.

Unless newlines in strings are escaped

Re: A love letter to the CSV format

#213

Post by solidsnack9000 »

mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

Tab-Separated Value, as implemented by many databases, solves these problems, because tab, newline and other control characters are escaped. For example, the default text serialization format of Postgres (`COPY TO ''` without any options) is this way.

Re: A love letter to the CSV format

#214

Post by benwilber0 »

gthompson512 wrote: post #186
benwilber0 wrote: post #114

Earlier quoted context omitted.

JSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and q…

> It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") You really super can't just split on commas for csv. You need to handle the string encodings since records can have commas occur in a string, and you need to handle quoting since you need to know when a string ends and that string may have internal quote characters. For either format unl…

Right, obviously.

Re: A love letter to the CSV format

#215

Post by kccqzy »

packetlost wrote: post #151
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

You're missing my point: basically nothing spits out data in that format because it's not ergonomic to do so. JSON is designed to represent object hierarchies, not tabular data.

JSON is designed to represent JavaScript objects with literal notation. Guess what, an array of strings or an array of numbers or even an array of mixed strings and numbers is a commonly encountered format in JavaScript.

Re: A love letter to the CSV format

#216

Post by Yomguithereal »

mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

A good way to parallelize CSV processing is to split datasets into multiple files, kinda like manual sharding. xan has a parallel command able to perform a wide variety of map-reduce tasks on splitted files.

https://github.com/medialab/xan

Re: A love letter to the CSV format

#217

Post by jandrese »

thesuitonym wrote: post #130
jandrese wrote: post #76

Earlier quoted context omitted.

If there were visible well known characters that could be printed for those and keys on a keyboard for inputting them we would probably have RSV files. Because they are buried down in the nonprintable section of the ASCII chart they are a pain for people to deal with. All it would have taken is one more key on the keyboard, maybe splitting the tab key in half.

> If there were visible well known characters that could be printed... ...There would be datasets that include those characters, and so they wouldn't be as useful for record separators. Look into your heart and know it to be true.

I wouldn't feel too bad about blindly scrubbing those characters out of inputs unlike commas, tabs, and quotes.

Re: A love letter to the CSV format

#219

Post by solidsnack9000 »

nayuki wrote: post #10

I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values

TSV's big advantage is that, as commonly implemented, the separators are escaped, not quoted. This means that a literal newline (ASCII 0x0A) is always a record separator and a literal tab (ASCII 0x09) is always a field separator. This is the format many databases -- including Postgres -- use for text export by default.

There are some notes I put together about TSV a few years ago that expand on these points: https://github.com/solidsnack/tsv?tab=readme-ov-file#motivat...

Re: A love letter to the CSV format

#220

Post by solidsnack9000 »

mort96 wrote: post #38
nayuki wrote: post #10

I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values

TSV looks incredibly ugly when opened in a text editor unless all values are 7 characters or less.

How does CSV look when the fields are all of different widths?
Post reply on HN