Live data from Hacker News

CSV 1.1 – CSV Evolved (for Humans)

csv11.github.io

51–60 of 81 posts

Re: CSV 1.1 – CSV Evolved (for Humans)

#51

There are a ton of problems here: * From the documentation "No quotes needed for values ... Use dual quotes ... Use triple quotes" * You haven't solved the problem of describing what the columns are. In fact, its worse because you are encouraging people to put units in the field * Spaces matter! What if the data is literally " Word " vs "Word". This format makes them both the same. * For some reason, the header row i…

Hello, I'm the original author of CSV 1.1.

There are no problems :-). You're making up things / problems.

Nowhere says it that the header row is removed. You CANNOT auto-detect a header, you have to tell your parser (see the csvreader library as an example - https://github.com/csv11/csvreader - from my humble self.) if you have a header or not (it's optional).

If Spaces matter! but them in quotes. By default you don't need quotes (and, thus, discouraged).

> You haven't solved the problem of describing > what the columns are

That is solved / done by a schema with a (tabular) datapackage, see https://github.com/csv11/csvpack as a real-world example how that works in practice or use csvrecord, see https://github.com/csv11/csvrecord

Re: CSV 1.1 – CSV Evolved (for Humans)

#52

CSV is hell, glad to see improvements. Some idiot somewhere decided that Comma Separated Values in certain locales should be based on semicolons (who would have thought files would be shared across country borders!?), so when we open CSV files that are actually comma separated all the information is in the first cell (until a semicolon appears). To get comma separated CSVs to show properly in Excel we have to mess ar…

Some idiots like... everyone except US and UK?

http://mathworld.wolfram.com/DecimalPoint.html

Re: CSV 1.1 – CSV Evolved (for Humans)

#54
post #2

I can't see this gaining any traction. There's nothing wrong with CSV as it is for basic data transfer. CSV had been unchanged now for so long, that any attempt to update it will fail - there are simply too many CSV parsers already implemented. If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields.

> If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields.

Good point. That's what (tabular) data packages are for, see https://github.com/csv11/csvpack for some real-world examples.

Re: CSV 1.1 – CSV Evolved (for Humans)

#55

I was extremely confused why the website nor the repository does not contain the specification. There is an informal specification littered in a separate repo [1], and the whole organization [2] is dedicated for the documentations and associated Ruby implementations. So well, good luck, but next time try to be more specific. [1] https://github.com/csv11/csv-next [2] https://github.com/csv11/

FYI: The readme (the website) is the specification :-), see https://github.com/csv11/csv11.github.io PS: csv-next is NOT an informal specification - these are notes (collection of ideas). PPS: How do I know? I'm the author of the website - I should know ;-).

Oh, hello! My gut feeling was that csv-next is pointing to the informal specification (and they read like it, but specific to a single problem point).

I guess you might not have understood my complaint (my bad), so let me give some examples how the specification should look like.

An informal specification looks like this [1]. You should give examples and rules enough to use your format and sufficient to implement most of the things. You should give definitions for keywords that are specific to your specification (but you can omit common definitions). You should give some (but probably not all) ideas where the specification can go wrong: Unicode whitespaces, Byte Order Mark, platform-specific newlines, escape sequences, duplicate keys in the front matter (well, actually there is no provision how to put the front matter at all), numeric overflows, and so on.

A formal specification looks like this [2]. In addition to what an informal specification provides, you should give lots of examples and formalized rules (most frequently ABNF [3]) to implement all the things. There should be clear and reasonable error handling policies. The wording of the specification should be clear, unambiguous and preferably standardized (there are specific meanings to "MUST", "SHOULD" etc. [4]). The specification should be honest about its pros and cons. You should be explicit about the flexibility of the format: you should give a list of what can be extended or modified later and what can't.

I strongly suggest you to put an informal specification at the least, and to prepare for the eventual development of a formal specification by pondering about missing pieces (it is a daunting task, I know).

[1] https://github.com/toml-lang/toml/blob/bb47759841ac368d86eb7...

[2] https://tools.ietf.org/html/rfc7049

[3] https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_...

[4] https://tools.ietf.org/html/bcp14

Re: CSV 1.1 – CSV Evolved (for Humans)

#56

For those looking for a modern take on improving on the CSV format, I'd recommend Frictionless Data's Datapackage specification[1] which basically consists of a json file of metadata that accompanies a CSV file that describes column types, versions, sources, and how to validate correctness of the CSV's data. This allows for quite a lot of tooling and workflow improvements to CSV files without mucking with the CSV its…

FYI: The (tabular) datapackage is great / fantastic but one layer up the stack.

See https://github.com/csv11/csvpack for real-world data package examples.

By the way, the tabular data csv dialect specification - is a great start/initiative (mostly a 1:1 copy from the python parser :-), really - would need an update, for more options, to reflect the reality of the csv formats out there. The big insight and breakthrough - csv is NOT one spec or format - but various flavors / formats / dialect - let the computer (that is, csvreader library) handle it.

Re: CSV 1.1 – CSV Evolved (for Humans)

#57
Good effort. And I like the minimal documentation. However, and I would love to be proved wrong, I think you are about 20 years late for this project to get any interest aside from some effimerous moments in hackernews...

Re: CSV 1.1 – CSV Evolved (for Humans)

#58
post #35

The best thing we have today for "CSV evolved" is JSON array of arrays, IMO. It's much better specified, everything agrees on how things should be quoted, it's perfectly human-readable if you do one array per line, its semantics follows straightforwardly from that of JSON, and we already have parsers for it that produce a reasonable output. Is it slightly more verbose? Sure, but why does it matter? Having to quote al…

Good point. See the CSVJSON format - http://csvjson.org - Love it. I will add a new variant called CSV <3 JSON shortly to the CSV 1.1 repo and csvreader etc. too.

Re: CSV 1.1 – CSV Evolved (for Humans)

#59

Why not use tab-separated values, you get improved readability and less quoting needed, or am I missing something?

See the Awesome CSV page, for the discussion about tabs vs commas - https://github.com/csv11/awesome-csv#tab

Yes, ideally the tab is perfect (no escape rules needed, etc). In practice you cannot tell if you're human if a tab is a space or a space is a tab and than you will get into trouble reading / writing your data etc. Anyways, both are great and needed and use tab2csv or csv2tab to convert or pipe (when using command line tools) :-)

Post reply on HN