Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

221–230 of 711 posts

Re: A love letter to the CSV format

#221

Post by solidsnack9000 »

recursive wrote: post #25
adzm wrote: post #18

Earlier quoted context omitted.

Agreed, much easier to work with, especially if you can guarantee no embedded tabs or newlines. Otherwise you end up with backslash escaping, but that's still usually easier than quotes.

It's not really easier than CSV if you can guarantee no commas or newlines.

Parsing escapes is easier than parsing quoted text with field and record separators embedded in it. Every literal newline or literal tab is a separator. One can jump to the thousandth record, for example, just by skipping lines, without looking into the contents.

Re: A love letter to the CSV format

#222

Post by lyu07282 »

Maro wrote: post #31

I hate CSV (but not as much as XML). Most reasonably large CSV files will have issues parsing on another system.

It makes me a bit worried to read this thread, I would've thought its pretty common knowledge why CSV is horrible and widely agreed upon. I also have hard time taking anybody seriously who uses "specification" and "CSV" in the same sentence unironically.

I suspect its 1) people who worked with legacy systems AND LIKED IT, or 2) people who never worked with legacy systems before and need to rediscover painful old lessons for themselves.

It feels like trying to convince someone, why its a bad idea to store the year as a CHAR(2) in 1999, unsuccessfully.

Re: A love letter to the CSV format

#223

Post by tfvlrue »

tengwar2 wrote: post #7

I'm not really sure why "Excel hates CSV". I import into Excel all the time. I'm sure the functionality could be expanded, but it seems to work fine. The bit of the process I would like improved is nothing to do with CSV - it's that the exporting programs sometimes rearrange the order of fields, and you have to accommodate that in Excel after the import. But since you can have named columns in Excel (make the data in…

In the past I remember that Excel not properly handling UTF-8 encoded text in a CSV. It would treat it as raw ASCII (or possibly code page 1252). So if you opened and saved a CSV, it would corrupt any Unicode text in the file. It's possible this has been fixed in newer versions, I haven't tried in a while.

Re: A love letter to the CSV format

#224

Post by otikik »

LoganDark wrote: post #200
notpushkin wrote: post #197

Earlier quoted context omitted.

Justifiable how?

Decimal separators being commas in some locales?

They could have just ignored the locale altogether though. Put dots on the numbers when using csv, and assume it has dots when importing

Re: A love letter to the CSV format

#225

Post by theLiminator »

nomel wrote: post #178
Someone1234 wrote: post #107

Earlier quoted context omitted.

For context: I have a LOT of experience of interchange formats, like "full time job, every day, all day, hundreds of formats, for 20-years" experience. Based on that experience I have come to one key, but maybe, counter-intuitive truth about interchange formats: - Too much freedom is bad . Why? Generating interchange data is cheaper than consuming it, because the creator only needs to consider the stuff they want to…

Oh, this is interesting. Are you tying different systems together? If so, do you use some preferred intermediate format? Do you have a giant library of * -> intermediate -> * converters that you sprinkle between everything? Or maybe the intermediate format is in memory? What about Parquet and the like?

For tabular/time-series greater than 100k rows I personally feel like parquet cannot be beat. It's self-describing, strongly-typed, relatively compact, supports a bunch of io/decode skipping, and is quite fast.

Orc also looks good, but isn't well supported. I think parquet is optimal for now for most analytical use-cases that don't require human readability.

Re: A love letter to the CSV format

#226

Post by lyu07282 »

circadian wrote: post #29

Kudos for writing this, it's always worth flagging up the utility of a format that just is what it is, for the benefit of all. Commas can also create fun ambiguity, as that last sentence demonstrates. :P CSV is lovely. It isn't trying to be cool or legendary. It works for the reasons the author proposes, but isn't trying to go further. I work in a work of VERY low power devices and CSV sometimes is all you need for a…

Because if there is anything we love in data exchange formats its ambiguity.

Re: A love letter to the CSV format

#227

Post by zzo38computer »

WorldMaker wrote: post #165
corysama wrote: post #141

Earlier quoted context omitted.

The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not…

In Windows (and DOS EDIT.COM and a few other similarly ancient tools) there have existed Alt+028, Alt+029, Alt+030, and Alt+031 for a long time. I vaguely recall some file format I was working with in QBASIC used some or all of them and I was editing those files for some reason. That was not quite as far back as 1984, but sometime in the early 1990s for sure. I believe EDIT.COM had basic glyphs for them too, but I do…

> I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits.

That is not specific to EDIT.COM; they are the PC characters with the same codes as the corresponding control characters, so they appear as graphic characters. (They can be used in any program that can use PC character set.)

However, in EDIT.COM and QBASIC you can also prefix a control character with CTRL+P in order to enter it directly into the file (and they appear as graphic characters, since I think the only control characters they will handle as control characters are tabs and line breaks).

Suits are PC characters 3 to 6; these are PC characters 28 to 31 which are other shapes.

Re: A love letter to the CSV format

#228

Post by zzo38computer »

LegionMammal978 wrote: post #173
corysama wrote: post #141

Earlier quoted context omitted.

The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not…

> The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." There's also the argument of "Now you have two byte values that cannot be allowed to appear in a record under any circumstances. (E.g., incoming data from uncontrolled sources MUST be sanitized to reject or repl…

One benefit of binary formats is not needing the escaping.

Re: A love letter to the CSV format

#229

Post by theLiminator »

k_bx wrote: post #156

I've recently been developing a raspberry pi based solution which works with telemetry logs. First implementation used an SQLite database (with WAL log) – only to find it corrupted after just couple of days of extensive power on/off cycles. I've since started looking at parquet files – which turned out to not be friendly to append-only operations. I've ended up implementing writing events into ipc files which then pe…

> I've since started looking at parquet files – which turned out to not be friendly to append-only operations. I've ended up implementing writing events into ipc files which then periodically get "flushed" into the parquet files. It works and it's efficient – but man is it non-trivial to implement properly!

I think the industry standard for supporting this is something like iceberg or delta, it's not very lightweight, but if you're doing anything non-trivial, it's the next logical move.

Post reply on HN