Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

261–270 of 594 posts

Re: Time to retire the CSV?

#261

Post by titzer »

qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

I, too, am a fan of ISO 8601. In an insane world of date formats, it's the only sane choice.

https://en.wikipedia.org/wiki/ISO_8601

Re: Time to retire the CSV?

#262

Post by joelthelion »

No love for parquet and/or feather?

They have open-source implementations in many languages, are much faster to load than csv, are natively compressed, are strongly typed and don't require parsing...

There are few reasons to continue using csv in this day and age.

Re: Time to retire the CSV?

#263

Post by lmilcin »

srcreigh wrote: post #116
lmilcin wrote: post #107

Earlier quoted context omitted.

SQLite also requires external dependency and special execution environment and are not as portable as CSV.

SQLite is included in most operating systems by default. It's also on macOS and iOS. It also runs on all kinds of embedded devices in addition to personal computers and servers. https://en.wikipedia.org/wiki/SQLite#Operating_systems

Just because most popular operating systems have libraries, doesn't yet mean it is portable.

Can I click on it and open it in Excel?

If not then it is not portable for me and for a lot of other people.

The main reason I use CSV is to produce reports that I can either open myself or send to other people so that they can click on it and open themselves and immediately start hacking away.

Excel is still corporate lingua franca and until this changes CSV is not going to retire.

CSV means for me that I can send it to anybody and everybody will know how to use it.

Try sending SQLite binary to people you work with, to your accountant, to your wife. Then share your experience.

Re: Time to retire the CSV?

#264

Post by karteum »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

I personally believe that at least SQLite matches all those criteria :

"1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...)

"2) Applications have a speed increase from using csvs" : I think it should be obvious to everyone that this is the case...

"3) The new format should not be grossly larger than the one it is replacing" : this is also the case

Re: Time to retire the CSV?

#265

Post by Traster »

CSV is ubiqitous. That's a massive advantage. If you argue against using CSV you might make it less ubiquitous. You're never going to make anything as ubiquitous as CSV though and that's amply demonsrated by the fact that when you look at alternatives you immediately name 7 different things. And some of the advantages aren't even necessarily advantages - a file format storing typed values is a non-trivial consideration when you're thinking about cross-tool support. Pandas is cursed with this even in CSV - there wasn't an optional int so any column with missing values must be float, and floats don't necessarily handle large integers - have fun with your nanoseconds going funny in your nanosecond timestamps stored as optional ints.

CSV is ubiqituous because it's not trying to solve difficult problems. The second you try to solve the difficult problems you necessarily fragment your audience.

Re: Time to retire the CSV?

#266

Post by p_j_w »

qayxc wrote: post #192
_moof wrote: post #127

This is an example of a genre I like to describe as: programmer objects to a solution that meets everyone's requirements because it doesn't make them feel like a beautiful code-poet. I like elegance as much as anyone. And I think it's a good proxy for other important qualities. But don't prioritize it above building something that actually does the job. Be an engineer.

Ah yes, and then there's https://www.theverge.com/2020/8/6/21355674/human-genes-renam... Not to mention the mess that is exchanging documents between different locales. It's all sunshine and roses until you get your CSVs from an office in a different country (which happens a lot in Europe). CSV gets the job done until it doesn't.

>CSV gets the job done until it doesn't.

Then you take it out of use where it doesn't get the job done.

Re: Time to retire the CSV?

#267

Post by mongol »

cogman10 wrote: post #251
mongol wrote: post #239

Earlier quoted context omitted.

Interesting. This is JSON++ somehow. What should it be called? Line-oriented JSON? Row-JSON?

It's just JSON. It's not an extension to the spec, it's a subset of the spec. If anything, you could say it's JSON--.

Well, to work as a CSV-like JSON, i.e greppable similar to a CSV it needs formatting with linebreaks as above. It is still completely valid JSON. ++ is the formatting convention.

Re: Time to retire the CSV?

#268

Post by cbsmith »

masklinn wrote: post #242
cbsmith wrote: post #212

Earlier quoted context omitted.

> The entire article is about replacing CSVs for exchanging data exported from Excel... No, it's not. It's about replacing CSVs for exchanging data. It mentions that CSVs often are the product of someone exporting data from a spreadsheet or doing a table dump, and how just doing that tends to create a ton of problems, but Excel is an example, not the subject matter of the article. > The business world will laugh at y…

> The business world pays me a lot of money to teach them not to use CSVs. Could you teach them better and faster? I don't think they're getting it. You have my blessing to use violence.

> Could you teach them better and faster? I don't think they're getting it. You have my blessing to use violence.

I'm trying man. I'm trying.

Re: Time to retire the CSV?

#269

Post by kstrauser »

qwerty456127 wrote: post #193
CobaltFire wrote: post #165

Earlier quoted context omitted.

Honestly I constantly see dates argued about and people state various formats that are still confusing. 4-2-2 of any variety can be confused. Why not 2-3-4 or 4-3-2 (DD-MMM-YYYY or YYYY-MMM-DD)? I’ve never understood why that isn’t more widely used.

> Why not ... DD-MMM-YYYY Because sorting. You can just sort a collection of dates stored as YYYY-MM-DD strings alphabetically and the result will always be in accordance with the actual time line.

This has so many happy side effects. Yesterday I was upgrading some code to use a newer version of an API. The old version stored timestamps as int seconds-since-epoch, but the new one stored them as ISO8601 strings. I replaced a line of code like:

  if row['timestamp']
with:

  if row['isotimestamp']
and everything kept working exactly as before. There was no expensive and error prone date parsing involved, some some simple string comparison. It wasn't quite as cheap as comparing ints, but it's not performance critical code so eh, good enough.

Re: Time to retire the CSV?

#270

Post by cbsmith »

da_chicken wrote: post #217
cbsmith wrote: post #204

Earlier quoted context omitted.

> That's only true if you're trying to send all your data in a single, monolithic CSV. No, that's true in general. There are exceptional cases where it might not be true, but it's true in general. > Yes, you do have situations where you're storing losing data density due to using plain text strings, but that's not a limitation particularly unique to CSV for data serialization formats. That there are other inefficient…

> No, that's true in general. There are exceptional cases where it might not be true, but it's true in general. Okay, put your money where your mouth is. Prove it.

CSV uses decimal representations of numeric data, which means you are getting 3.5 bits of data for every 8 bits of storage space (and that's assuming you are using a reasonably compact text encoding... if you are using UTF-16, it's 16 bits). Using a binary representation you can store 8 bits of data for every 8 bits of storage space.

CSV uses a variety of date-time formats, but a prevalent one is YYYY-MM-DDThh:mm:ss.sssZ. I'll leave it as an exercise for the reader to determine whether that is as compact as an 8-byte millis since the epoch value.

CSV also requires escaping of separator characters, or quoting of strings (and escaping of quotes), despite ASCII (and therefore UTF-8) having a specific unit separator character already reserved. So you're wasting space for each escape, and effectively wasting symbol space as well (and that's ignoring the other bits of space for record separators, group separators, etc.).

Then there's floating point data...

Honestly, this is a silly argument. CSV is, by design, a format that trades space for other factors. It should not be debatable whether it is a compact format or not.

Post reply on HN