Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

331–340 of 594 posts

Re: Time to retire the CSV?

#331

Post by zabzonk »

pessimizer wrote: post #214
zabzonk wrote: post #61

As the author of a CSV munging tool (CSVfix) I think most of the problems with CSV could be fixed if people producing CSV output, and people reading CSV input obeyed the rules of the RFC. Sadly, most people don't, and any textual output or input is routinely described as CSV, when it is nothing of the sort - even to the extent of not being comma-separated!

If you're referring to http://csvfix.byethost5.com/csvfix15/csvfix.html , thank you from the bottom of my heart. You were my pick for best tool for dealing with CSVs when a bunch of others (some of which sadly ended up with better adoption) were competing for the prize.

Yep, that's me. I'm afraid, because of mental and physical health problems, and family troubles I have stopped supporting it. Also, when bitbucket stopped supporting Mercurial I simply could not be arsed porting it to git. Not that I have anything against git.

Re: Time to retire the CSV?

#332

Post by hermitcrab »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically

Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't store Tabs or carriage returns in your data. Also there is no way to store metadata in TSV.

JSON is far from ideal for storing 2D data tables as it is a tree. This means it is much more verbose than it needs to be. The same is also true for XML.

Re: Time to retire the CSV?

#333

Post by dredmorbius »

mongol wrote: post #252
pharmakom wrote: post #195

What are good alternatives to CSV that provide all of these features: - easy to parse - easy to edit with a generic text editor - easy to edit with a widely available GUI, like LibreOffice - allow adding more data with only append operations

I recall ESR's book about Unix conventions and culture mentioned something I think he called stanza-based. A file format where each line is a value and records separated by empty lines. Don't think Office tools can work with that though, except as regular text files.

Awk will eat that for breakfast no problem.

The usual problem is when you discover that what you thought was a value is in fact a record of its own, with internal structure.

The problem isn't the CSV. The problem is that data parsing is actually a Hard Problem.

The author of TFA is ... naive and misguided.

Re: Time to retire the CSV?

#334

Post by lufte »

WorldMaker wrote: post #310
lufte wrote: post #295

Earlier quoted context omitted.

Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end. How would any existing JSON parser handle 75GB of data in a single array of arrays?

Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not. JSON.Parse() in…

Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.

Re: Time to retire the CSV?

#335

Post by derefr »

cbsmith wrote: post #321
wombatpm wrote: post #285

Earlier quoted context omitted.

TSV solve a lot of the pain

...or, you know, you could use the ASCII characters specifically defined for separating records and units. ;-)

If only I could type them on my keyboard. (I think this is a big part of why CSV is the way it is — people want to be able to hand-edit it, or at least hand-produce small test datasets to test the systems on the other end.)

Re: Time to retire the CSV?

#336

Post by kcartlidge »

turtlebits wrote: post #315
nightski wrote: post #130

Earlier quoted context omitted.

So you can constrain what type of CSV you will allow and if this happens it will bail. It's that simple. There is nothing wrong with having additional constraints on top of just saying it must be "CSV" especially in these scenarios. I'm in a similar situation, we've been using CSV for over a decade to move billions of dollars worth of product each year. It just works.

I'm pretty sure most devs are going to use whatever CSV library that comes with their language. When that breaks, it's generally not a simple fix.

> I'm pretty sure most devs are going to use whatever CSV library that comes with their language. When that breaks, it's generally not a simple fix.

Call me a yak-shaver, but in every language I've worked with I've written my own csv parsing library when I needed one.

It's such a trivial thing for the majority cases (varying of delimiters, line-endings, ascii/uft8, quoting, escaping, and embedding of delimiters/line-endings) that it takes barely no time at all after you've done it once in another language. Of course there are edge cases and special cases depending upon specific workloads, but if your team has it's own parser (which is a small amount of obvious code) then it does indeed usually become a simple fix.

Sounds good using someone else's library, but below a certain complexity it's rarely worth it in the medium to long term except for when doing proof of concept or demo code, or if the domain space is complex.

Re: Time to retire the CSV?

#337

Post by derefr »

asdff wrote: post #237
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

Those other things have different names like TSV

And then Excel imports them all using heuristics when you select "Type: CSV". So you'll never train anyone on the demand end of these documents that they're called anything besides CSVs.

Re: Time to retire the CSV?

#338

Post by hermitcrab »

wombatpm wrote: post #288
cbsmith wrote: post #212

Earlier quoted context omitted.

> The entire article is about replacing CSVs for exchanging data exported from Excel... No, it's not. It's about replacing CSVs for exchanging data. It mentions that CSVs often are the product of someone exporting data from a spreadsheet or doing a table dump, and how just doing that tends to create a ton of problems, but Excel is an example, not the subject matter of the article. > The business world will laugh at y…

Of course there is a old solution in the ANSI character set. File, Record, Group and Unit separator characters

Yes. You could get a long way with a text format in which:

-the first line is always a header

-fields are separated by Unit separator characters

-records are separated by Record separator characters

-encoding is UTF8

If you wanted to get fancy you could also have:

-comment lines

-column metadata (e.g. column 0 is an ISO date, column 2 is text, column 3 is an integer)

Both the above could start with a Unicode character unlikely to be used for anything else.

I think that would avoid 99% of the pain of CSV files. The downside is that the use of things like the Unit separator mean that it wouldn't be easy to create/edit manually.

I don't suppose it will ever happen though.

Re: Time to retire the CSV?

#339

Post by hermitcrab »

wombatpm wrote: post #285
cbsmith wrote: post #212

Earlier quoted context omitted.

> The entire article is about replacing CSVs for exchanging data exported from Excel... No, it's not. It's about replacing CSVs for exchanging data. It mentions that CSVs often are the product of someone exporting data from a spreadsheet or doing a table dump, and how just doing that tends to create a ton of problems, but Excel is an example, not the subject matter of the article. > The business world will laugh at y…

TSV solve a lot of the pain

As long as you don't need to store tabs or carriage returns in your data. ;0)

Re: Time to retire the CSV?

#340

Post by jojo2000 »

It's time to reinvent the wheel, but more complex, more failure-prone. No thanks ! Joke aside good alternatives exist, but are far less ubiquitous. Binary format means problems to read, compatibility issues... Look at how hdf5 is a complex format with a lengthy spec. Checksum and compress your csv files, it will solve most issues.
Post reply on HN