Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

1–10 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#3

Post by cogman10 »

CSVs are bad. If you can change the format then don't use a DSV, use parquet and a library for your language to consume parquet.

It's less code for you and you can do neat things like zstd compression on the columns.

Bonus, it also doesn't require that you load and unload everything in memory.

https://arrow.apache.org/docs/python/parquet.html

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#5

Post by mjevans »

This keeps coming up as new people discover what CSVs are. An ancient TEXT data exchange format. The lowest vaguely common denominator. A style of format with flavors software long out of support contract are happy to export data in.

The intent of the format is to be human readable and editable. Sure, Tab characters can be used instead of commas. (TSV files) Yes that's that "" to escape a quote rule. Oh and quoted values are optional, unquoted strings are fine as long as they contain no newline or record separator characters.

Sure, you could make another CSV inspired format which uses the old mainframe control characters; except as keeps getting pointed out, even programmers often don't know how to enter raw flow control characters on their systems. Who even bashes those out these days? I know I have to look it up every time.

Rejoice that the format is so simple, it's all just text which software might convert to numbers or other values as it might desire.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#6

Post by teddyh »

The article talks about reading and parsing CSV data of unknown variants, but then skips to the solution being using a different format altogether. But you can only switch to a different format if you are producing data, not if you are reading it!

And if you are in control of producing data, just produce strict RFC 4180-compliant CSV data and everybody will be able to read it just fine. There is no need to make your reader’s lives difficult by using yet another non-standard data format.

See also: https://news.ycombinator.com/item?id=39679753>

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#7

Post by wmal »

The author seems to ignore the fact that CSV got so popular because it is human readable. If anyone wanted a binary format there’s plenty of them - most better than this DSV.

Also, I’m on a mobile right now, so can’t verify that, but it seems the format is flawed. The reader decodes UTF8 strings after splitting the binary buffer by the delimiter, but I believe the delimiter may be a part of a UTF8 character.

Edit: just checked and there’s actually no chance that the delimiter the author chose would be part of UTF8 encoding of any other character than the delimiter itself

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#9

Post by Pikamander2 »

From what I've seen, the biggest problem isn't with the CSV standard (even though it has a lot of faults), but rather that a lot of software that utilizes CSVs is poorly tested.

I can't tell you how many times I've downloaded a CSV that didn't escape quotes or newlines correctly, or how many times Excel has failed to correctly parse a perfectly valid CSV due to some decades-old quirk.

I know that there are better formats that make these types of situations pop up less, but is a little bit of quality control too much to ask for? If you've tested your software to make sure that it can handle CSVs with line breaks, tabs, and both types of quotes, then you've seemingly done more testing than 90% of the software out there.

On that note, the LibreOffice Calc team deserves major credit for how many different CSV formats it can handle. It's saved my bacon so many times when Excel wasn't up to the task.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#10

Post by NoboruWataya »

The only real benefit of CSV (other than that it is widely supported) is that it is easy for humans to read and write. The approach in this article solves the quoting problem, but also removes that benefit. If you have the power to move from CSV, surely JSON would be better if you need to keep the human readable/writable feature. And if you don't need it, there are other more featureful binary formats out there like parquet.
Post reply on HN