CSV 1.1 – CSV Evolved (for Humans) • Hacker News Board

Re: CSV 1.1 – CSV Evolved (for Humans)

#41

Post by infogulch » Fri, Oct 12, 2018, 6:37 AM UTC

Someone1234 wrote: post #17

Jaruzel wrote: post #2

I can't see this gaining any traction. There's nothing wrong with CSV as it is for basic data transfer. CSV had been unchanged now for so long, that any attempt to update it will fail - there are simply too many CSV parsers already implemented. If you are worried about people in the year 3000 understanding your data, add a .txt file alongside the CSV explaining the fields.

I agree. The biggest problem with CSV is Excel's terrible support. And while CSV is primarily used for data transfer, Excel could hypothetically be a great debug-viewer if it didn't do brain dead stuff like unrecoverably corrupt UPCs and other long numbers by default. There's an Excel Uservoice about this very issue from 2015: https://excel.uservoice.com/forums/304921-excel-for-windows-... They're "looking into it."…

Excel's braindead csv corruption is why I never use it for csv now. Add guids to the list of things it can corrupt. (if the first group is numeric it parses as an expression which then causes an error because the next group has hex digits, what!?) Also any long string that happens to only be digits. And two numbers separated by a "/".

It's so bad I check any csv file I get if it's been opened in Excel, and if I find any evidence that it has I reject it and make whoever I got it from give me the original source, or transform it with literally any other tool.

Re: CSV 1.1 – CSV Evolved (for Humans)

#42

Post by clatan » Fri, Oct 12, 2018, 6:38 AM UTC

Yeah... No. This doesn't solve anything.

Re: CSV 1.1 – CSV Evolved (for Humans)

#43

Post by stephenr » Fri, Oct 12, 2018, 6:39 AM UTC

legostormtroopr wrote: post #13

There are a ton of problems here: * From the documentation "No quotes needed for values ... Use dual quotes ... Use triple quotes" * You haven't solved the problem of describing what the columns are. In fact, its worse because you are encouraging people to put units in the field * Spaces matter! What if the data is literally " Word " vs "Word". This format makes them both the same. * For some reason, the header row i…

Edit to add: your main points are spot on, IMO, but:

Original: I'm not even sure I agree that 'comments' are good, particularly if they're going to be intentionally ignored by GUI s/s apps.

If you have meaningful information, put it in the furthest right-column, so it appears in spreadsheet editors on the right of the data.

If it's not meaningful to the person who will open the file, why include it?

Re: CSV 1.1 – CSV Evolved (for Humans)

#44

Post by kalleboo » Fri, Oct 12, 2018, 6:40 AM UTC

Reminds me of old IBM punch card fixed-column-aligned tables... Still in use in many systems (such as the US ACH wire system...)

Everything old is new again.

Re: CSV 1.1 – CSV Evolved (for Humans)

#45

Post by cpeterso » Fri, Oct 12, 2018, 7:04 AM UTC

ASCII defines control codes for record and field separators, thus avoiding CSV's problems with quoting and escaping characters. The only hassle is entering and displaying the control characters.

https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_...

Re: CSV 1.1 – CSV Evolved (for Humans)

#46

Post by webdevetc » Fri, Oct 12, 2018, 7:17 AM UTC

While it is nice to add some spacing and make things a bit easier to read, I don't really see the point in this.

If you want it to be easily parsable by a human then there are hundreds of applications designed for this. The most common being spreadsheets like Excel.

And what happens if you have a list of domains, but one contains a really long url (300 characters wide?). It will mess up the columns for every single row above and below it as well.

Also adding comments will completely break any existing software (I think most programs can handle some more whitespace, but you couldn't import the final example into Excel, I don't think (untested))

Re: CSV 1.1 – CSV Evolved (for Humans)

#47

Post by petepete » Fri, Oct 12, 2018, 7:27 AM UTC

Tempest1981 wrote: post #9

romanovcode wrote: post #6

I even don't see how it is better. ID,Name,Capital,Area,Tags bc,British Columbia,Victoria,922509,en|western canada vs ########################### # Oh, Canada! 10 provinces and 3 territories # # see en.wikipedia.org/wiki/Provinces_and_territories_of_Canada # # note: key is two-letter canadian postal code # # for regions tags see # en.wikipedia.org/wiki/List_of_regions_of_Canada bc, British Columbia, Victoria, 922 509…

Right, where are the tag names specified? What if a new column is added, and my reader needs to handle both formats? Also, seems like specification-by-example, unless I’m missing something.

To be honest, I'd be quite happy if CSV just had a 'frontmatter' block. Sometimes a bit of background info or description of what the data represents or its source is really useful.

Removing column headings would be a killer. Although efforts in the past to add type information are misguided, by that point you may as well just use XML.

Re: CSV 1.1 – CSV Evolved (for Humans)

#48

Post by bachmeier » Fri, Oct 12, 2018, 7:44 AM UTC

You can already do this in R with a fixed-width text file, comments, and the names of variables in a header. The commas serve no actual purpose AFAICT.

Re: CSV 1.1 – CSV Evolved (for Humans)

#49

Post by geraldbauer » Fri, Oct 12, 2018, 7:49 AM UTC

lifthrasiir wrote: post #16

I was extremely confused why the website nor the repository does not contain the specification. There is an informal specification littered in a separate repo [1], and the whole organization [2] is dedicated for the documentations and associated Ruby implementations. So well, good luck, but next time try to be more specific. [1] https://github.com/csv11/csv-next [2] https://github.com/csv11/

FYI: The readme (the website) is the specification :-), see https://github.com/csv11/csv11.github.io

PS: csv-next is NOT an informal specification - these are notes (collection of ideas).

PPS: How do I know? I'm the author of the website - I should know ;-).

Re: CSV 1.1 – CSV Evolved (for Humans)

#50

Post by robochat42 » Fri, Oct 12, 2018, 7:58 AM UTC

_pwalsh wrote: post #30

ewheeler wrote: post #10

For those looking for a modern take on improving on the CSV format, I'd recommend Frictionless Data's Datapackage specification[1] which basically consists of a json file of metadata that accompanies a CSV file that describes column types, versions, sources, and how to validate correctness of the CSV's data. This allows for quite a lot of tooling and workflow improvements to CSV files without mucking with the CSV its…

One of the authors of the Frictionless Data specifications here. The spec directly relevant to CSV is Table Schema [1], and we’ve also got some nice tools that leverage Table Schema and the family of specifications, such as goodtables [2]. [1] https://frictionlessdata.io/specs/table-schema/ [2] https://github.com/frictionlessdata/goodtables-py

This looks interesting and useful. It might be a feature that wouldn't get used a lot but I've always felt that it would be great to have a 'unit' field to specify the physical units of the values. Have you ever discussed adding this as an optional field to the specification?