Time to retire the CSV? • Hacker News Board

Re: Time to retire the CSV?

#591

Post by CRConrad » Thu, Sep 02, 2021, 6:14 AM UTC

Dylan16807 wrote: post #584

edmundsauto wrote: post #376

Earlier quoted context omitted.

Yep, you got me. I'm actually a trapeze artist moonlighting on these forums. A CSV file represents the exact digits that are to be stored. You have unlimited precision. You could even store irrational numbers, equations, or mix data types in a column. OTOH, you have to make sure the delimiting character is not present in the data - that can be pretty easy, if you use the field-separator character in ASCII, or even ju…

Excel isn't a database.

Which is exactly what GP didn't write.

"A bird, like homo sapiens, has two feet" doesn't mean that birds are humans or humans are birds either. It means that in this respect, birds and humans are alike. Which is what GP meant: With respect to conforming data, databases behave just like Excel.

Language note: I think it's the fact that there are two commas around the inserted-clause [terminology?] ", like Excel," that does it. If there were only one comma, before or after "like Excel" it would read the way you read it. Can be tricky for non-native (and sometimes also native) speakers.

Re: Time to retire the CSV?

#592

Post by CRConrad » Thu, Sep 02, 2021, 6:32 AM UTC

cbsmith wrote: post #557

da_chicken wrote: post #416

Earlier quoted context omitted.

> CSV uses decimal representations of numeric data, which means you are getting 3.5 bits of data for every 8 bits of storage space (and that's assuming you are using a reasonably compact text encoding... if you are using UTF-16, it's 16 bits). Using a binary representation you can store 8 bits of data for every 8 bits of storage space. XML, JSON, and YAML all have this issue, too. > CSV uses a variety of date-time fo…

> XML, JSON, and YAML all have this issue, too. Yes. Though to their credit, some of those work with her numbers, which at least gets you 4 bits out of every 8 bits. > And I know what you're about to argue, but JSON's datetime format is not in the spec. The common JSON datetime format is convention, not standard. I'm not sure what argument you thought I was making, or why that comment is relevant. > All you've shown…

> some of those work with her numbers

Whose?

> The claim was that it was a compact format for data transfer, which is demonstrably not true.

More compact than the realistic competitors (XML, JSON, YAML, etc).

Re: Time to retire the CSV?

#593

Post by CRConrad » Thu, Sep 02, 2021, 8:08 AM UTC

pwinnski wrote: post #582

AussieWog93 wrote: post #404

Earlier quoted context omitted.

>If you think you can do better than CSV, let's see your proposal. I've got one! It's basically the same as regular CSV, but everything is UTF-8, the columns and lines are delineated by dedicated UTF-8 "delineator" codepoints (if they aren't defined in the spec, find reasonable surrogates and use them), and therefore nothing ever needs to be escaped. More human readable than regular CSV, less prone to error and just…

So... I can't just type up a file in this new format in a text editor, since the dedicated low-byte delineator characters don't have corresponding keys on my keyboard. That makes it not quite as good as CSV for at least one use case, and certainly much less flexible. Replacing flexible standards with more-rigid standards is always really hard.

> the dedicated low-byte delineator characters don't have corresponding keys on my keyboard

[Alt]-[(NumPad:)0031]

[Alt]-[(NumPad:)0030]

Yeah, not quite dedicated keys. But once you've typed in one of each, you can copy and paste. Somewhat better text editors hopefully display them in some fashion (e.g, NotePad++ shows them as little black blobs labelled "FS" / "GS" / "RS" / "US").

Re: Time to retire the CSV?

#594

Post by pwinnski » Wed, Sep 08, 2021, 5:32 PM UTC

CRConrad wrote: post #593

pwinnski wrote: post #582

Earlier quoted context omitted.

So... I can't just type up a file in this new format in a text editor, since the dedicated low-byte delineator characters don't have corresponding keys on my keyboard. That makes it not quite as good as CSV for at least one use case, and certainly much less flexible. Replacing flexible standards with more-rigid standards is always really hard.

> the dedicated low-byte delineator characters don't have corresponding keys on my keyboard [Alt]-[(NumPad:)0031] [Alt]-[(NumPad:)0030] Yeah, not quite dedicated keys. But once you've typed in one of each, you can copy and paste. Somewhat better text editors hopefully display them in some fashion (e.g, NotePad++ shows them as little black blobs labelled "FS" / "GS" / "RS" / "US").

That's a solution for MS Windows computers with extended keyboards only, so still not on my keyboard. The set of keyboards that lack numpads or are not connected to machines running MS Windows is much larger than the set of keyboards without commas.

But yes, nothing is technically insurmountable. It's just hard to imagine a format that requires such efforts overtaking a format that does not.