Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

101–110 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#101

Post by [deleted] »

zelphirkalt wrote: post #20
NoboruWataya wrote: post #10

The only real benefit of CSV (other than that it is widely supported) is that it is easy for humans to read and write. The approach in this article solves the quoting problem, but also removes that benefit. If you have the power to move from CSV, surely JSON would be better if you need to keep the human readable/writable feature. And if you don't need it, there are other more featureful binary formats out there like…

JSON is often good, but it also has potentially a lot of overhead, depending on how sparse the data is. For sparse data, it might be better. But for not sparse data, it will have the overhead of mentioning attribute names over and over again. Of course you could also have arrays in JSON, not writing attribute names over and over, but then you are basically back to a CSV inside the JSON file ...

[deleted]

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#102

Post by solarkraft »

atrus wrote: post #29
throw0101d wrote: post #24

Earlier quoted context omitted.

[flagged]

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

“Just be in control” is unfortunately bad advice when you’re … not in control.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#103

Post by zelphirkalt »

imtringued wrote: post #30
zelphirkalt wrote: post #20

Earlier quoted context omitted.

JSON is often good, but it also has potentially a lot of overhead, depending on how sparse the data is. For sparse data, it might be better. But for not sparse data, it will have the overhead of mentioning attribute names over and over again. Of course you could also have arrays in JSON, not writing attribute names over and over, but then you are basically back to a CSV inside the JSON file ...

>Of course you could also have arrays in JSON, not writing attribute names over and over, but then you are basically back to a CSV inside the JSON file ... You're confusing the concept of tabular data with the file format. If the most natural way to represent tabular data is through a 2D array, then so be it. The vast majority of people aren't complaining about the fact that they have to hardcode the meaning of "the…

Where am I confusing the two?

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#104

Post by brunokim »

usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

"Broken" is a sliding scale, and it's unfeasible to refuse engaging at all times.

If you are a multi-billion dollar company creating a new integration, you can demand that your small supplier provide an RFC-4180 compliant file, and even refuse to process it if its schema or encoding is not conformant.

If you are the small supplier of a multi-billion dollar company, you will absolutely process whatever it is that they send you. If it changes, you will even adapt your processes around it.

TFA proposes a nice format that is efficient to parse and in some ways better than CSV, another ways are not. Use it if you can and makes sense.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#105

Post by brunokim »

tbrownaw wrote: post #32

> If we used 31 as a field delimiter and 30 instead of newlines, we solve every single edge case from above. Why? Because these are non-printing characters that should never appear in a text-stream data set. I have in fact seen CSV files used as an interchange format for things that include non-plaintext fields. And I've seen nested CSV files.

LOL nested CSVs are a new one to me. What was it used for?

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#106

Post by TheCleric »

solarkraft wrote: post #102
atrus wrote: post #29

Earlier quoted context omitted.

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

“Just be in control” is unfortunately bad advice when you’re … not in control.

I don't think the advice is "Just be in control" as much as "Acknowledge you're not in control most of the time."

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#107

Post by mattewong »

I cannot imagine any way it is worth anyone's time to follow this article's suggestion vs just using something like zsv (https://github.com/liquidaty/zsv, which I'm an author of) or xsv (https://github.com/BurntSushi/xsv/edit/master/README.md) and then spending that time saved on "real" work

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#108

Post by postalrat »

wmal wrote: post #7

The author seems to ignore the fact that CSV got so popular because it is human readable. If anyone wanted a binary format there’s plenty of them - most better than this DSV. Also, I’m on a mobile right now, so can’t verify that, but it seems the format is flawed. The reader decodes UTF8 strings after splitting the binary buffer by the delimiter, but I believe the delimiter may be a part of a UTF8 character. Edit: ju…

I'd say CSVs stuck around because there weren't any other alternatives that could be easily created, appended to, read by different apps.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#109

Post by ahmeneeroe-v2 »

throw0101d wrote: post #24
teddyh wrote: post #6

The article talks about reading and parsing CSV data of unknown variants, but then skips to the solution being using a different format altogether. But you can only switch to a different format if you are producing data, not if you are reading it! And if you are in control of producing data, just produce strict RFC 4180-compliant CSV data and everybody will be able to read it just fine. There is no need to make your…

[flagged]

As someone who has excel open all-day every-day: I don't think excel-employees are "producing" data. We are ingesting CSVs/XLSXs, performing modeling/analysis, and then saving as an XLSX.

I don't know anyone who is saving a CSV unless it is the final model output and another system (e.g. TM1) can only ingest the CSV.

If I accidentally save a spreadsheet as a CSV it is a bad day since I probably lost all my formatting, formulas, and additional tabs.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#110

Post by ahmeneeroe-v2 »

Freak_NL wrote: post #42
throw0101d wrote: post #24

Earlier quoted context omitted.

[flagged]

Importing data generated from Excel: don't, or force RFC 4180-compliant CSV data. Exporting data into Excel: provide RFC 4180-compliant CSV data, or just generate minimal XLSX files. Most Excel users generally don't export to CSV (or practice any data sanity conventions); they seem to believe XLSX is a perfectly fine data exchange format for automated use. (“Oh the import broke? I just added an empty row above the he…

>Those that do understand that automated data processing means you need to be stricter in what you do in the sheet you are exporting tend to understand how to export proper CSV's from Excel as well

exactly right

Post reply on HN