Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

61–70 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#61

Post by usrbinbash »

snapcaster wrote: post #55
usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

No it isn't in the real world. It's very much your problem if you're the team consuming these files. Try to go tell the head of accounting they need to make all their data rfc4180 compliant see how that goes

> Try to go tell the head of accounting they need to make all their data rfc4180 compliant see how that goes

Fun fact: I did. And not just for accounting systems, but all sorts of data ingestion pipelines. Did it work every time? No. Did it work in many cases? Yes. Is that better? Absolutely.

Here is the thing: If I accept broken CSV, where do I stop? What's next? Next thing my webservice backends have to accept broken HTTP? My JSON-RPC backends have to accept JSON with /*/ style block comments? My ODBC load-balancer has to accept natural language instead of SQL statements (I mean, its the age of the LLM, I could make that possible).

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#62

Post by welcome_dragon »

Ah yes good old CSV. It's perfectly fine to use for data transfer and there are libraries for (probably) every language that handle it perfectly to spec.

The problem isn't "CSV". The problems come from: - "excel doesn't like this CSV therefore it's not valid" - "what do you mean the CSV I sent you is wrong? I used excel" - "standard? What's a standard? I put info then a comma. That should be good enough for anyone"

CSV, when done right (i.e. following a standard) is a great format. It's human readable, less verbose than, say, JSON, and everybody understands it.

Just have to make sure business people (and less technical technical people) understand that CSV != Excel and vice-versa.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#63

Post by SigmundurM »

usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

> Don't accept broken CSV. If people keep ignoring standards, thats their problem.

From the very memo you link to (RFC 4180):

> Implementors should "be conservative in what you do, be liberal in what you accept from others" (RFC 793 [8]) when processing CSV files.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#64

Post by gsck »

beardyw wrote: post #60

I wanted a quick and dirty to parse a CSV in js the other day and just added square brackets around it and used JSON.parse. Am I alone in this?

That's actually pretty smart.

The amount of times I have written a simple CSV parser to correctly handle quoted strings and the like is more times than I have digits when I could just pretend its JSON.

Going to make a mental note to try this next time!

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#65

Post by RaftPeople »

vaylian wrote: post #58
lazide wrote: post #39

Earlier quoted context omitted.

How many get edited or inspected in notepad at some point in their life? Nearly all of them (for any given workflow).

It is nice that text editors are abundantly available and that they can be used for the task. But once the CSV columns get too wide and irregular, then you probably want to reach for a dedicated spreadsheet program, because it is otherwise too hard to figure out which column you are currently reading. There is still room between a text editor and a full-blown spreadsheet program. New DSV editors could emerge when the…

> when the DSV format gains popularity

CSV is equivalent to Voyager I, the chances of catching up with that kind of head start are extremely low.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#66

Post by gsck »

The phones we use at {JOB} can be programmatically controlled by using their proprietary command language, which is just CSV and each command is ended with a new line (Because how else would you do it, packet boundary pfft?).

It's something I've never understood why, why not use something more standard like SIP, or even a more structured message format. Having to parse CSV across N different packet boundaries is a royal PITA

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#67

Post by usrbinbash »

SigmundurM wrote: post #63
usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

> Don't accept broken CSV. If people keep ignoring standards, thats their problem. From the very memo you link to (RFC 4180): > Implementors should "be conservative in what you do, be liberal in what you accept from others" (RFC 793 [8]) when processing CSV files.

Oh, I am nothing but liberal when it comes to CSV: Clients get the liberty to either have their requests processed, or get a 400 BAD REQUEST

And yes, I am aware that the standard says this. My counter question to that is: How much client-liberty do I have to accept? Where do I draw the line? How much is too much liberty?

And the answer is: there is no answer. Wherever any system draws that line, it's an arbitrary decision; Except for one, which ensures the least surprise and maximum interoperability (aka. the point of a standard): to be "conservative", and simply demand the standard.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#68

Post by eth0up »

Question: I started with a deliberately convoluted PDF which after much effort I filtered, sorted, reorganized and transferred the 18000 useful lines to a csv. These lines are simple, with dates, indicator and corresponding numbers.

The purpose is to statically analyze the numbers for anomalies or any signs of deviation from expected randomness. I do this all in python3 with various libraries. It seems to be working, but...

What is a more efficient format than csv for this kind of operation?

Edit: I have also preserved all leading zeros by conversion to strings -- csv readers don't care much for leading zeros and simply disappear them, but quotes fix that.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#70

Post by usrbinbash »

dghf wrote: post #54
usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

> https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like Mm, not really. By its own admission, it is descriptive, not prescriptive : > This section documents the format that seems to be followed by most implementations And it came out in 2005, by which date CSVs had already been in use for some twenty or thirty years.

It doesn't matter when it came out, it doesn't matter that it it descriptive. It is the standard, period.

Yes, CSV is much, much older. In fact it predates personal computers. And it went through changes. Again: None of that matters. We have a standard, we should use the standard, and systems should demand the standard.

Standards are meant to ensure minimal-friction interoperability. If systems don't enforce standards, then there is no point in having a standard in the first place.

Post reply on HN