Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

561–570 of 711 posts

Re: A love letter to the CSV format

#561

Post by collinmanderson »

simonw wrote: post #154
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

Right - the JSON-newline equivalent of CSV can look like this: ["id", "species", "nickname"] [1, "Chicken", "Chunky cheesecakes"] [2, "Dog", "Wagging wonders"] [3, "Bunny", "Hopping heroes"] [4, "Bat", "Soaring shadows"]

This is the way. jsonl where each row is a json list. It has well-defined standard quoting.

Just like csv you don't actually need the header row either, as long as there's convention about field ordering. Similar to proto bufs, where the field names are not included in the file itself.

Re: A love letter to the CSV format

#562

Post by [deleted] »

hadlock wrote: post #543
kevincox wrote: post #515

Earlier quoted context omitted.

> and does not assign any significance to the ordering of name/value pairs. I think this is outdated? I believe that the order is preserved when parsing into a JavaScript Object. (Yes, Objects have a well-defined key order. Please don't actually rely on this...)

[flagged]

[deleted]

Re: A love letter to the CSV format

#563

Post by marcosdumay »

sgarland wrote: post #538
jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

> Waiting for someone to write a love letter to the infamous Windows INI file format... Honestly, it’s fine. TOML is better if you can use it, but otherwise for simple applications, it’s fine. PgBouncer still uses INI, though that in particular makes me twitch a bit, due to discovering that if it fails to parse its config, it logs the failed line (reasonable), which can include passwords if it’s a DSN string.

Well, once you get over the fact that information on a TOML file can be out of order in any place, denominated by any mix of 3 different key encodings, and broken down in any random way... then yes, the rest of TOML is good.

Re: A love letter to the CSV format

#564

Post by remram »

danso wrote: post #533
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

> Then there will be your data, but they will format currency values as prettified strings, for example "34 593,12 USD", instead of producing one column with a number and another with currency. To be fair, that's not a problem with CSV but with the provider's lack of data literacy.

Yeah, you can also use Parquet/JSON/protobuf/XLSX and store numbers as strings in this format. CSV is just a container.

Re: A love letter to the CSV format

#565

Post by PaulHoule »

kec wrote: post #532
PaulHoule wrote: post #526

Earlier quoted context omitted.

Accurate data typing (never confuse a string with a number) Maybe be circular but: always loads correctly into Excel, if you want to load into a spreadsheet you can add text formatting and even formulas, checkboxes and stuff which can be a lot of fun.

That is very much not true, Excel does type coercion, especially around things that happen to look like dates: https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Excel does that type coercion if you import from CSV. If you export pandas data to XLSX it adds proper type information and then it imports properly into Excel and you avoid those problems.

Re: A love letter to the CSV format

#566

Post by mrweasel »

raxxorraxor wrote: post #547
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

Not sure if they still do this, but Klarna would send us ", " separated files. If there wasn't a space after the comma then it was to be read as a decimal point. Most of the CSV parser don't/didn't allow you to specify multi-character separators. In the end I just accepted that we had one field for krona and for öre and most fields would need to have a leading space removed.

Re: A love letter to the CSV format

#567

Post by pasc1878 »

taeric wrote: post #540
pasc1878 wrote: post #504

Earlier quoted context omitted.

Same is true of CSV/TSV.

I think you are a bit more likely to notice in a CSV/TSV, as it is unlikely to truncate at a newline? Still, fair point. And is part of why I said it is a flaw, not the flaw. Plenty of other reasons to not like YAML, to me. :D

Not if it is split at a line e.g. if the source or target can only deal with a fixed number of lines.

Re: A love letter to the CSV format

#568

Post by marcosdumay »

999900000999 wrote: post #160
Xelbair wrote: post #140

Earlier quoted context omitted.

until you find someone abusing XSD schemas, or someone designing a "dynamically typed" XML... or sneaks in extra data in comments - happened to me way often than it should.

My condolences. Any open standard runs the risk of this happening. It's not a problem I think we'll ever solve.

In principle, if you make your standard extensible enough, people should stop sneaking data into comments or strings.

... What makes the GP's problem so much more amusing. XML was the last place I'd expect to see it.

Re: A love letter to the CSV format

#569

Post by imtringued »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

I agree and as a result I have completely abandoned CSV.

I use the industry standard that everyone understands: ECMA-376, ISO/IEC 29500 aka .xlsx.

Nobody has any problems producing or ingesting .xlsx files. The only real problem is the confusion between numbers and numeric text that happens when people use excel manually. For machine to machine communication .xlsx has never failed me.

Re: A love letter to the CSV format

#570

Post by edoceo »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

This RFC maybe?

https://www.ietf.org/rfc/rfc4180.txt

Post reply on HN