Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

481–490 of 711 posts

Re: A love letter to the CSV format

#481

Post by oever »

niccl wrote: post #280
dietr1ch wrote: post #251

Earlier quoted context omitted.

I don't understand why CSV became a thing when TSV, or a format using the nowadays weird ASCII control characters like start/end of text, start of heading, horizontal/vertical tab, file/group/record/unit separator. It seems many possible designs would've avoided the quoting chaos and made parsing sort of trivial.

Any time you have a character with a special meaning you have to handle that character turning up in the data you're encoding. It's inevitable. No matter what obscure character you choose, you'll have to deal with it

It's evitable by stating the number of bytes in a field and then the field. No escaping needed and faster parsing.

Re: A love letter to the CSV format

#482

Post by oever »

zzo38computer wrote: post #228
LegionMammal978 wrote: post #173

Earlier quoted context omitted.

> The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." There's also the argument of "Now you have two byte values that cannot be allowed to appear in a record under any circumstances. (E.g., incoming data from uncontrolled sources MUST be sanitized to reject or repl…

One benefit of binary formats is not needing the escaping.

Hear hear! Why is all editing done with text-based editors where humans can make syntax errors. Is it about job security?

Re: A love letter to the CSV format

#483

Post by immibis »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

XML is a pretty good markup language. Using XML to store structured data is an abuse of it. All the features that are useful for markup are not useful for structured data and only add to the confusion.

Re: A love letter to the CSV format

#485

Post by immibis »

mikepurvis wrote: post #47
mjevans wrote: post #28

Earlier quoted context omitted.

The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.

As soon as you give those characters magic meanings then suddenly people will have reason to want to use them— it'll be a CSV containing localization strings for tooltips that contain that character and bam, we'll be back to escaping. Except the usages of that character will be rare and so potentially way more scary. At least with quotes and commas, the breakages are everywhere so you confront them sooner rather than…

Graphical representations of the control characters begin at U+2400 in the "Control Pictures" Unicode block. Instead of the actual U+001E Record Separator, you put the U+241E Symbol for Record Separator in the help text.

Re: A love letter to the CSV format

#486

Post by immibis »

LegionMammal978 wrote: post #314
zzo38computer wrote: post #228

Earlier quoted context omitted.

One benefit of binary formats is not needing the escaping.

Length-delimited binary formats do not need escaping. But the usual "ASCII Delimited Text" proposal just uses two unprintable bytes as record and line separators, and the signalling is all in-band. This means that records must not contain either of those two bytes, or else the format of the table will be corrupted. And unless you're producing the data yourself, this means you have to sanitize the data before adding i…

At least you don't need these values in your data, unlike the comma, which shows up in human-written text.

If you do need these values in your data, then don't use them as delimiters.

Something the industry has stopped doing, but maybe should do again, is restricting characters that can appear in data. "The first name must not contain a record separator" is a quite reasonable restriction. Even Elon Musk's next kid won't be able to violate that restriction.

Re: A love letter to the CSV format

#487

Post by tucnak »

samdung wrote: post #400

CSV works because CSV is understood by non technical people who have to deal with some amount of technicality. CSV is the friendship bridge that prevents technical and non technical people from going to war. I can tell an MBA guy to upload a CSV file and i'll take care of it. Imagine i tell him i need everything in a PARQUET file!!! I'm no longer a team player.

This is so relatable to all data eng people from SWE background!

Thanks

Re: A love letter to the CSV format

#488

Post by billyzs »

hajile wrote: post #320
BeFlatXIII wrote: post #300

Earlier quoted context omitted.

What do you like so much about the pipe?

I think pipe is better too. Typical latin fonts divide characters into three heights: short like "e" or "m", tall like "l" or "P" and deep like "j" or "y". As you may notice, letters only use one or two of these three sections. Pipe is unique in that it uses all three at the same time from the very top to the very bottom. No matter what latin character you put next to it, it remains distinct. This makes the separator…

I already prefer using pipe as separator in logging; now you're telling me there is a chance that my logs can be automatically ingested as tabular data? Sign me up for this branch of the multiverse :)

Re: A love letter to the CSV format

#489

Post by julik »

Something I support completely - previously https://news.ycombinator.com/item?id=35418933#35438029

If CSV is indeed so horrible - and I do not deny that there can be an improvement - how about the clever data people spec out a format that

Does not require a bizarre C++ RPC struct definition library _both_ to write and to read

Does not invent a clever number encoding scheme that requires native code to decode at any normal speed

Does not use a fancy compression algorithm (or several!) that you need - again - native libraries to decompress

Does not, basically, require you be using C++, Java or Python to be able to do any meaningful work with it

It is not that hard, really - but CSV is better (even though it's terrible) exactly because it does not have all of these clever dependency requirements for clever features piled onto it. I do understand the utility of RLE, number encoding etc. I do not, and will not, understand the utility of Thrift/Avro, zstandard and brotli and whatnot over standard deflate, and custom integer encoding which requires you download half of Apache Commons and libboost to decode. Yes, those help the 5% to 10% of the use cases where massive savings can be realised. It absolutely ruins the experience for the other 90 to 95.

But they also give Parquet and its ilk a very high barrier of entry.

Post reply on HN