Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

491–500 of 711 posts

Re: A love letter to the CSV format

#491

Post by tim333 »

samdung wrote: post #400

CSV works because CSV is understood by non technical people who have to deal with some amount of technicality. CSV is the friendship bridge that prevents technical and non technical people from going to war. I can tell an MBA guy to upload a CSV file and i'll take care of it. Imagine i tell him i need everything in a PARQUET file!!! I'm no longer a team player.

Indeed the my main use is most financial services will output your records in csv, although I mostly open that in excel which sometimes gets a bit confused.

Re: A love letter to the CSV format

#492

Post by no_wizard »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

INI was for a long time a seemingly preferable format in the Python community for configuration for a long time, as I recall it.

Haven’t been a full time Python dev in sometime though, it seems TOML has supplanted that, but I remember thinking how interesting it was that Python had a built in INI parser and serializer

Re: A love letter to the CSV format

#493

Post by boogheta »

lyu07282 wrote: post #222
Maro wrote: post #31

I hate CSV (but not as much as XML). Most reasonably large CSV files will have issues parsing on another system.

It makes me a bit worried to read this thread, I would've thought its pretty common knowledge why CSV is horrible and widely agreed upon. I also have hard time taking anybody seriously who uses "specification" and "CSV" in the same sentence unironically. I suspect its 1) people who worked with legacy systems AND LIKED IT, or 2) people who never worked with legacy systems before and need to rediscover painful old less…

Maybe just read the love letter?

Re: A love letter to the CSV format

#494

Post by bell-cot »

Foobar8568 wrote: post #401
samdung wrote: post #400

CSV works because CSV is understood by non technical people who have to deal with some amount of technicality. CSV is the friendship bridge that prevents technical and non technical people from going to war. I can tell an MBA guy to upload a CSV file and i'll take care of it. Imagine i tell him i need everything in a PARQUET file!!! I'm no longer a team player.

Among the shit I have seen in CSV, no " for strings, including those with a return char, innovative SEP, date, numbers, no escape for " within strings, rows related to the reporting tools used to export to CSV etc

True. But most of those problems are pretty easy for the non-technical person to see, understand, and (often) fix. Which strengthens the "friendship bridge".

(I'm assuming the technical person can easily write a basic parsing script for the CSV data - which can flag, if not fix, most of the format problems.)

For a dataset of any size, my experience is that most of the time & effort goes into handling records which do not comply with the non-technical person's beliefs about their data. Which data came from (say) an old customer database - and between bugs in the db software, and abuse by frustrated, lazy, or just ill-trained CSR's, there are all sorts of "interesting" things, which need cleaning up.

Re: A love letter to the CSV format

#495

Post by jellyfishbeaver »

I work as a data engineer in the financial services industry, and I am still amazed that CSV remains the preferred delivery format for many of our customers. We're talking datasets that cost hundreds of thousands of dollar to subscribe to.

"You have a REST API? Parquet format available? Delivery via S3? Databricks, you say? No thanks, please send us daily files in zipped CSV format on FTP."

Re: A love letter to the CSV format

#496

Post by pasc1878 »

boogheta wrote: post #454
IanCal wrote: post #445

Earlier quoted context omitted.

The byte for a capital I is the same as the start for an odd file format, slyk maybe? Excel has (or did if they finally fixed it) for years decided this was enough to assume the file (called .csv) cannot possibly be csv but must actually be slyk. It then parses it as such, and is shocked to find your slyk file is totally broken!

It sounds to me like as often the problem here is Excel, not CSV

Yes but in practice CSV is defined by what Excel does.

As there is no standard to which Excel conforms as it predates standards and there would be an outcry if Excel started rejecting files that had worked for years.

Re: A love letter to the CSV format

#497

Post by dspillett »

IanCal wrote: post #403
lelanthran wrote: post #396

Earlier quoted context omitted.

To be honest, I'm wondering why you are rating JSON higher than CSV. > Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, There is, actually, RFC 4180 IIRC. > there are many flavours that are incompatible with each other in the sense that a reader for one flavour would not be suitable for reading the other and vice versa. "There are many flavours that deviate from the s…

Have you had to work with csv files from the wild much? I'm not being snarky but what you're talking about is night and day to what I've experienced over the years. There aren't vast numbers of different JSON formats. There's practically one and realistically maybe two. Headers are in each line, utf8 has never been an issue for me and quoting and escaping are well defined and obeyed. This is because for datasets, alm…

WRT JSON:

> Headers are in each line

This might be my old “space and network cost savings” reflex, which is a lot less necessary these days, kicking in, but the feels inefficient. It also gives rise to not knowing the whole schema until you read the whole dataset (which might be multiple files), unless some form of external schema definition is provided.

Having said that, I accept that JSON has advantages over CSV, even if all that is done is translating a data-table into an array of objects representing one row each.

> utf8 has never been an issue for me

The main problem with UTF8 isn't with CSV generally, it is usually, much like the “first column is called ID” issue, due to Excel. Unfortunately a lot of people interact with CSVs primarily with Excel, so it gets tarred with that brush by association. Unless Excel sees the BOM sequence at the start of a CSV file, which the Unicode standards recommend against for UTF8, it assumes its characters are using the Win1252 encoding (almost, but not quite, ISO-8859-1).

> Csv files have all kinds of separators

I've taken to calling them Character Separated Value files, rather than Comma, for this reason.

Re: A love letter to the CSV format

#498

Post by pasc1878 »

afiori wrote: post #438
mastax wrote: post #389

Earlier quoted context omitted.

ASCII has characters more or less designed for this 0x1C - File Separator 0x1D - Group Separator 0x1E - Record Separator 0x1F - Unit Separator So I guess 1F would be the "comma" and 1E would be the "newline."

https://stackoverflow.com/questions/8695118/what-are-the-fil... I am pretty sure you shifted the meaning, the decimal separator is part of the atomic data it does not need a control character. You would use 1F instead of the comma/semicolon/tab and 1E to split lines (record means line just like in SQL). You could then use 1D to store multiple CSV tables in a single file.

Yes but then the text is not human readable or editable in a plain text editor.

This would confuse most users of csvs they are not programmers they at most use text editors and Excel.

Re: A love letter to the CSV format

#499

Post by realitysballs »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

They do specifically mention this:

“No one owns CSV. It has no real specification (yes, I know about the controversial ex-post RFC 4180), just a set of rules everyone kinda agrees to respect implicitly. It is, and will forever remain, an open and free collective idea.”

Re: A love letter to the CSV format

#500

Post by pasc1878 »

gthompson512 wrote: post #186
benwilber0 wrote: post #114

Earlier quoted context omitted.

JSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and q…

> It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") You really super can't just split on commas for csv. You need to handle the string encodings since records can have commas occur in a string, and you need to handle quoting since you need to know when a string ends and that string may have internal quote characters. For either format unl…

Yes but people don't
Post reply on HN