Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

581–590 of 711 posts

Re: A love letter to the CSV format

#581

Post by raxxorraxor »

mrweasel wrote: post #566
raxxorraxor wrote: post #547

Earlier quoted context omitted.

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

Not sure if they still do this, but Klarna would send us ", " separated files. If there wasn't a space after the comma then it was to be read as a decimal point. Most of the CSV parser don't/didn't allow you to specify multi-character separators. In the end I just accepted that we had one field for krona and for öre and most fields would need to have a leading space removed.

Microsoft did this very extensively. Many Non-English versions of Excel do save CSV-files with a semicolon as a separator and it probably was handled differently too in normal Excel files. But it goes even further, it affected their scripting languages even to this day with newer languages like their BI script (forgot the name of the language). For example, parameters of function calls aren't separated by ',' anymore and ';' is used instead. But only in the localized versions.

That of course means that you have to translate these scripts depending on the locale set in your office suite, otherwise they are full of syntax errors...

Re: A love letter to the CSV format

#582

Post by imtringued »

evnp wrote: post #138
Suppafly wrote: post #102

Earlier quoted context omitted.

That mostly breaks down to "excel is intentionally stupid with csv files if you don't use the import function to open them" along with the normal "don't trust customer input without stripping or escaping it" concerns you'd have with any input.

That was my initial reaction as well – it's a vulnerability in MS software, not ours, not our problem. Unfortunately, reality quickly came to bear: our customers and employees ubiquitously use excel and other similar spreadsheet software, which exposes us and them to risk regardless where the issue lies. We're inherently vulnerable because of the environment we're operating in, by using CSV. "don't trust customer inp…

If your customers and employees are using Excel then stop going against the grain with your niche software developer focused formats that need a lot of explanations.

I need to interface with a lot of non-technical people who exclusively use Excel. I give them .xlsx files. It's just as easy to export .xlsx as it is to export .CSV and my customers are happy.

Re: A love letter to the CSV format

#583

Post by abought »

At various points in my career, I've had to oversee people creating data export features for research-focused apps. Eventually, I instituted a very simple rule:

As part of code review, the developer of the feature must be able to roundtrip export -> import a realistic test dataset using the same program and workflow that they expect a consumer of the data to use. They have up to one business day to accomplish this task, and are allowed to ask an end user for help. If they don't meet that goal, the PR is sent back to the developer.

What's fascinating about the exercise is that I've bounced as many "clever" hand-rolled CSV exporters (due to edge cases) as other more advanced file formats (due to total incompatibility with every COTS consuming program). All without having to say a word of judgment.

Data export is often a task anchored by humans at one end. Sometimes those humans can work with a better alternative, and it's always worth asking!

Re: A love letter to the CSV format

#584

Post by imtringued »

togakangaroo wrote: post #206
evnp wrote: post #138

Earlier quoted context omitted.

That was my initial reaction as well – it's a vulnerability in MS software, not ours, not our problem. Unfortunately, reality quickly came to bear: our customers and employees ubiquitously use excel and other similar spreadsheet software, which exposes us and them to risk regardless where the issue lies. We're inherently vulnerable because of the environment we're operating in, by using CSV. "don't trust customer inp…

Hey, I'm the author of the linked article, cool to see this is still getting passed around. Definitely agree there's no perfect solution. There's some escaping that seems to work ok, but that's going to break CSV-imports. An imperfect solutions is that applications should be designed with task-driven UIs so that they know the intended purpose of a CSV export and can make the decision to escape/not escape then. Librar…

>Another imperfect solution would be to ... ugh...generate exports in Excel format rather than CSV. I know, I know, but it does solve the problem.

Or you could just use the ISO standard .xlsx, which is a widely supported format that is not Excel specific but has first class support in Excel.

Re: A love letter to the CSV format

#585

Post by zzo38computer »

LegionMammal978 wrote: post #314
zzo38computer wrote: post #228

Earlier quoted context omitted.

One benefit of binary formats is not needing the escaping.

Length-delimited binary formats do not need escaping. But the usual "ASCII Delimited Text" proposal just uses two unprintable bytes as record and line separators, and the signalling is all in-band. This means that records must not contain either of those two bytes, or else the format of the table will be corrupted. And unless you're producing the data yourself, this means you have to sanitize the data before adding i…

I did mean length-delimited binary formats (rather than ASCII formats).

Re: A love letter to the CSV format

#586

Post by Sohcahtoa82 »

gghffguhvc wrote: post #199
lgas wrote: post #182

Earlier quoted context omitted.

Typing isn't optional in JSON, every value has a concrete type, always.

[ [“header1”,”header2”], [“1.1”, “”], [7.4, “2022-01-04”] ]

...and?

I see an array of arrays. The first and second arrays have two strings each, the last one has a float and a string. All those types are concrete.

Let's say those "1.1" and 7.4 values are supposed to be version strings. If your code is only sometimes putting quotes around the version string, the bug is in your code. You're outputting a float sometimes, but a string in others. Fix your shit. It's not your serialization format that's the problem.

If you have "7.4" as a string, and your serialization library is saying "Huh, that looks like a float, I'm going to make it a float", then get a new library, because it has a bug.

Re: A love letter to the CSV format

#587

Post by jszymborski »

deathanatos wrote: post #260
jszymborski wrote: post #159

Earlier quoted context omitted.

I like ndjson and jsonl just fine, but unless I need a more complicated structure, it's not worth the extra hassle of parsing JSON.

… what concrete language are we talking about, here? In literally any language I can think of, hassle(json) < hassle(CSV), esp. since CSV received is usually "CSV, but I've screwed it up in a specific, annoying way"

I'm thinking mostly of the computational complexity.

But even ergonomically, in python, can read a csv like:

  import csv
  [row for row in csv.DictReader(f)]
which imo is not less ergonomic than

  import json
  [json.loads(line) for line in f]

Re: A love letter to the CSV format

#588

Post by JadeNB »

Moru wrote: post #552
raxxorraxor wrote: post #547

Earlier quoted context omitted.

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

There was a long period of my life that I thought .csv meant cemicolon separated because all I saw was cemicolon separated files and I had no idea of the pain.

Although it is spelled "semicolon," so that doesn't quite fit.

Re: A love letter to the CSV format

#589

Post by Sami_Lehtinen »

raxxorraxor wrote: post #547
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

TSV should do it for you. Been there done that.

Re: A love letter to the CSV format

#590

Post by SoftTalker »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.
Post reply on HN