Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

571–580 of 711 posts

Re: A love letter to the CSV format

#571

Post by [deleted] »

mjw_byrne wrote: post #531
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

I used to be a data analyst at a Big 4 management consultancy, so I've seen an awful lot of this kind of thing. One thing I never understood is the inverse correlation between "cost of product" and "ability to do serialisation properly". Free database like Postgres? Perfect every time. Big complex 6-figure e-discovery system? Apparently written by someone who has never heard of quoting, escaping or the difference bet…

[deleted]

Re: A love letter to the CSV format

#572

Post by imtringued »

pasc1878 wrote: post #496
boogheta wrote: post #454

Earlier quoted context omitted.

It sounds to me like as often the problem here is Excel, not CSV

Yes but in practice CSV is defined by what Excel does. As there is no standard to which Excel conforms as it predates standards and there would be an outcry if Excel started rejecting files that had worked for years.

There is a common misconception here. You can import CSV files into an excel sheet. You cannot open a CSV file with excel. That is a nonsense operation.

Re: A love letter to the CSV format

#573

Post by Eridrus »

akritid wrote: post #375
Eridrus wrote: post #355

Earlier quoted context omitted.

Doing sequential reading into a queue for workers to read is a lot more complicated than having a file format that supports parallel reading. And the fix to allow parallel reading is pretty trivial: escape new lines so that you can just keep reading until the first unescaped new line and start at that record. It is particularly helpful if you are distributing work across machines, but even in the single machine case,…

The practical solution is to generate several CSV files and distribute work at the granularity of files

Sure, now you need to do this statically ahead of time.

It's not unsolvable, but now you have a more complicated system.

A better file format would not have this problem.

The fix is also trivial (escape new lines into \n or similar) would also make the files easier to view with a text editor.

Re: A love letter to the CSV format

#574

Post by afiori »

boogheta wrote: post #468
afiori wrote: post #446

Earlier quoted context omitted.

Actually even whitespace-separated json would be a valid format and if you forbid json documents to be a single integer or float then even just concatenating json gives a valid format as JSON is a prefix free language. That is[0] if a string s is a valid JSON then there is no substring s[0..i] for i So you could just consume as many bytes you need to produce a json and then start a new one when that one is complete.…

« a single missing " can destroy the entire document » This is basically true for any data format, so really worse argument ever...

In jsonl a modified chunk will lose you at most the removed lines and the two adjacent ones (unless the noise is randomly valid json), in particular a single byte edit can destry at most 2 lines.

utf-8 is also similarly self-correcting and so is html and many media formats.

My point was that in my made-up concatenated json format

[]"""[][][][][][][][][][][]"""[]

and

[]""[][][][][][][][][][][]""[]

are both valid but have differ only for 2 bytes but have entirely different structures.

Also it is a made-up format nobody uses (if somebody were to want this they would likely disallow strings at the root level).

Re: A love letter to the CSV format

#575

Post by larusso »

I can‘t really understand the love for the format. Yes it’s simple but also not defined in a common spec. Same story with markdown. Yes GitHub tried to push for a spec but it still feels more like a flavor. I mean there is nothing wrong with not having a spec. But certain guarantees are not given. Will the document exported by X work with Y.

Re: A love letter to the CSV format

#576

Post by taeric »

pasc1878 wrote: post #567
taeric wrote: post #540

Earlier quoted context omitted.

I think you are a bit more likely to notice in a CSV/TSV, as it is unlikely to truncate at a newline? Still, fair point. And is part of why I said it is a flaw, not the flaw. Plenty of other reasons to not like YAML, to me. :D

Not if it is split at a line e.g. if the source or target can only deal with a fixed number of lines.

Right, that is what I meant about that being unlikely? Most instances of truncated files that I have seen were because of size, not lines.

Still, a fair point.

Re: A love letter to the CSV format

#577

Post by esbranson »

CSV on the Web (CSVW) is a W3C standard designed to enable the description of CSV files in a machine-readable way.[1]

"Use the CSV on the Web (CSVW) standard to add metadata to describe the contents and structure of comma-separated values (CSV) data files." — UK Government Digital Service[2][3]

[1] https://www.w3.org/TR/tabular-data-primer/

[2] https://www.gov.uk/government/publications/recommended-open-...

[3] https://csvw.org/

Re: A love letter to the CSV format

#578

Post by watwut »

lelanthran wrote: post #414
IanCal wrote: post #403

Earlier quoted context omitted.

Have you had to work with csv files from the wild much? I'm not being snarky but what you're talking about is night and day to what I've experienced over the years. There aren't vast numbers of different JSON formats. There's practically one and realistically maybe two. Headers are in each line, utf8 has never been an issue for me and quoting and escaping are well defined and obeyed. This is because for datasets, alm…

Sure, I get your arguments and we're probably mostly in agreement, but in practice I see very few problems arising with using CSV. I mean, right now , the data interchange format between multiple working systems is CSV; think payment systems, inter-bank data interchange, ERP systems, CRM systems, billing systems ... the list goes on. I just recently had a coffee with a buddy who's a salesman for some enterprise syste…

> I mean, right now, the data interchange format between multiple working systems is CSV; think payment systems, inter-bank data interchange, ERP systems, CRM systems, billing systems ... the list goes on.

And there are constant issues arising from that. You basically need a small team to deal with them in every institution that is processing them.

> I just recently had a coffee with a buddy who's a salesman for some enterprise system: of the most common enterprise systems we recently worked with (SAP type things, but on smaller scales), every single one of them had CSV as the standard way to get data between themselves and other systems.

Salesman of enterprise system do not care about issues programmers and clients have. They care about what they can sell to other businessmen. That teams on both sides then waste time and money on troubleshooting is no concern to the salesman. And I am saying that as someone who worked on the enterprise system that consumed a lot of csv. It does not work and process of handling them literally sometimes involved phone calls to admins of other systems. More often then would be sane.

> The number of people uploading excel files to be processed or downloading excel files for local visualistation and processing would floor you.

That is perfectly fine as long as it is a manager downloading data so that he can manually analyze them. It is pretty horrible when those files are then uploaded to other systems.

Re: A love letter to the CSV format

#579

Post by imtringued »

samdung wrote: post #400

CSV works because CSV is understood by non technical people who have to deal with some amount of technicality. CSV is the friendship bridge that prevents technical and non technical people from going to war. I can tell an MBA guy to upload a CSV file and i'll take care of it. Imagine i tell him i need everything in a PARQUET file!!! I'm no longer a team player.

This is incorrect. Everyone uses Excel, not CSV. There are billions of people on this planet who know what to do with an .xlsx file.

Do the same with a .CSV file and you'll have to teach those people how to use the .CSV importer in Excel and also how to set up the data types for each column etc. It's a non trivial problem that forces you down to a few million people.

.CSV is a niche format for inexperienced software developers.

Re: A love letter to the CSV format

#580

Post by [deleted] »

Moru wrote: post #552
raxxorraxor wrote: post #547

Earlier quoted context omitted.

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

There was a long period of my life that I thought .csv meant cemicolon separated because all I saw was cemicolon separated files and I had no idea of the pain.

[deleted]
Post reply on HN