Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

231–240 of 594 posts

Re: Time to retire the CSV?

#231

Post by alexpw »

1) CSV can often be human readable at a glance (more so than most other formats, depending on that data), and that makes it appear deceptively simple and compact. Possibly due to that, I'd bet most of us have been bit by a writer/reader that doesn't respect the RFC rules.

2) I ask for TSV, whenever convenient. It's been more reliable, and I don't have a comprehensive why, but I think it's slightly more resilient to writer/reader inconsistencies, for me. It may be that there's just less need for escaping and quoting, so you might dodge a smart quotes debacle when asking for a one-off from an Excel user, for example.

3) Despite the issues raised, the notion we'd retire it makes me hug it tight, because for the majority of my requirements, it hits a sweet spot. I still reserve the right to raise my fist in frustration when someone does: ",".join(mylist).

Re: Time to retire the CSV?

#232

Post by jrm4 »

Ha. Not gonna happen; for the same reason Python took off over more "flexible" languages, for the same reason people move from JSON to YAML for configs, etc.

Languages and formats are not fundamentally about computers or efficiency, they're about people. Carry on.

Re: Time to retire the CSV?

#233

Post by glogla »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Avro and Parquet and Arrow are almost loseless - they can't handle timestamp with timezone which really sucks for processing world-wide data.

I'm still looking for somwthing that can do it.

(Of course, Spark doesn't support timestampz which is probably why the formats don't.)

Re: Time to retire the CSV?

#234

Post by masklinn »

edmundsauto wrote: post #200
masklinn wrote: post #164

Earlier quoted context omitted.

> No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. SQLite's "dynamic data types" coerce data on input, not output. Once the data is in sqlite the way you wanted it, excel has no interpretation to perform, except insofar as really really wanting dates. > The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Yeah…

How is a csv not the most accurate representation of the data? If you trust the other agent encoded it properly in the db, then sure. Your flippant dismissal was inappropriate in tone and detracted from the rest of your opinion. Cockiness tells me that you’re insecure about your knowledge, not that you know more than GP.

> How is a csv not the most accurate representation of the data? If you trust the other agent encoded it properly in the db, then sure.

The idea that a CSV would be more likely to be correctly encoded than a DB is hilarious, thanks for the laugh. But that you were confident enough to seriously put it in writing shows how little experience you have with CSV.

Re: Time to retire the CSV?

#235

Post by X6S1x6Okd1st »

edmundsauto wrote: post #200
masklinn wrote: post #164

Earlier quoted context omitted.

> No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. SQLite's "dynamic data types" coerce data on input, not output. Once the data is in sqlite the way you wanted it, excel has no interpretation to perform, except insofar as really really wanting dates. > The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Yeah…

How is a csv not the most accurate representation of the data? If you trust the other agent encoded it properly in the db, then sure. Your flippant dismissal was inappropriate in tone and detracted from the rest of your opinion. Cockiness tells me that you’re insecure about your knowledge, not that you know more than GP.

There's plenty of CSVs that have been produced or will be parsed by

for line in input: ','.join(line)

It's not exactly a problem with "CSV" specifically, but the environment in which it exists.

Re: Time to retire the CSV?

#236

Post by Macha »

dragonwriter wrote: post #224
Macha wrote: post #221

Earlier quoted context omitted.

So someone opens this CSV in Excel and there's garbage in A1? Does this really count as compatible? You will get user bugs for this.

> So someone opens this CSV in Excel and there's garbage in A1? Yeah, that's why I chose the “thing that looks like a text file—including optionally CSV—but has additional metadata after the EOF mark” approach instead of stuffing additional metadata in the CSV; there's no way to guarantee that existing implementations will safely ignore any added metadata the main CSV body. (My mechanism has some risk in that there a…

If by EOF char you mean Ctrl-Z, Python's `csv` module is at least one case where it will read past the EOF char and you'll get rows of garbage data for any content in the file after that.

Re: Time to retire the CSV?

#237

Post by asdff »

gsich wrote: post #158
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

Those other things have different names like TSV

Re: Time to retire the CSV?

#238

Post by systemvoltage »

radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

100%. Don’t fix if it ain’t broken. This is the pragmatic approach that often comes across to new devs as “unsexy”. I got some news for “unsexy” software - it works, and it brings the revenue.

Re: Time to retire the CSV?

#239

Post by mongol »

cogman10 wrote: post #135
shapefrog wrote: post #113

Earlier quoted context omitted.

Thats just a CSV with extra steps.

Nope, that's CSV without the drawbacks of CSV. That's CSV that can have special characters and doesn't suffer from delimiter problems. When someone says "Maybe we can fix CSV" this is what you should do instead of trying to "fix" CSV.

Interesting. This is JSON++ somehow. What should it be called? Line-oriented JSON? Row-JSON?

Re: Time to retire the CSV?

#240

Post by laichzeit0 »

Lack of types is my biggest gripe with CSV. I think just being able to specify types in column headers would be a win. E.g. “column1:int,column2:string,column3:datetime”. Type inference has bit me too many times.
Post reply on HN