Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

221–230 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#221

Post by zzo38computer »

clarkevans wrote: post #183
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.

Do you have the file format documentation?

Re: Why isn’t there a decent file format for tabular data?

#222

Post by zzo38computer »

Banana699 wrote: post #192
civilized wrote: post #165

Earlier quoted context omitted.

Code may be data but data isn't code.

Data is code that prints/evaluates to itself.

In some programming languages (such as PostScript, where evaluating as itself vs being executed, is a flag separate from the value's type, and can be changed by the cvlit and cvx operators), it is.

Re: Why isn’t there a decent file format for tabular data?

#223

Post by zzo38computer »

asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

I suppose that another possible format is PostScript format, which has some of its own advantages and disadvantages. It has both text and binary format, and the binary object sequence format can easily be read by a C code (I wrote a set of macros to do so) without needing to parse the entire structure into memory at first; just read the element that you need, alone, one at a time, in any order. (I have used this format in some of my own software.)

JSON also has its own advantages and disadvantages.

Of course, both of these formats are more structured than CSV, but XML is also a more structured format.

(And, like I and others have mentioned too, the format that they dsecribed isn't actually unique; it just doesn't seem to be that common, but there are enough people who had done the same thing independently, and a few programs which support it, that you could use it if you want to do, and it should work OK.)

Re: Why isn’t there a decent file format for tabular data?

#225

Post by tech2 »

mbreese wrote: post #156
prepend wrote: post #58

Earlier quoted context omitted.

I prefer commas because I can see them over tabs. I spend zero time escaping commas because the libraries and read and write with (usually pandas but pretty much everything) escape for me. So unless I’m manually building csvs myself it’s a non-issue and certainly not histrionics.

I prefer tabs because my data can often have commas in it. Seeing tabs isn’t an issue as I also have invisible characters visible in all of my editors. But having your delimiter not be allowed in the record (as in \t is disallowed), makes parsing so much easier. CSV is a bear to parse because you have to read each value from a buffer to handle the quoting. line.strip().split(‘\t’) Is so handy. Batteries included are…

Given your example looks very much like python you could use the CSV module which is built into the standard library and handles all of this for you in a standards compliant manner.

You really do want to use a library to parse CSV since there are a number of corner cases. For example, your example code may not read a whole row since rows can have newlines in them.

Re: Why isn’t there a decent file format for tabular data?

#226

Post by zzo38computer »

Pxtl wrote: post #26

> Why can’t we have a format where Does Excel support it? No? Then that's the end of that. Excel is tabular data to all non developers. The formats supported by Excel are the whole thing. And if we're inventing a CSV-like format that uses a more convenient character than quotes and commas, maybe jumping to a non-displayable non-typeable character isn't the best? Honestly, if I were inventing a table format, I'd use a…

Is used anywhere else?

(I had considered ; my reason was that it seems to work better with user CSS, and might be simpler to parse in other cases too.)

Re: Why isn’t there a decent file format for tabular data?

#227

Post by d--b »

1. Either you want to edit the file in a text editor and you need to have a separator that has a key on your keyboard. Comma is indeed not the greatest choice, but tab or vertical pipe are usually fine. Most parsers will allow you to define the separator.

2. Or you don’t and you’re much better off with any binary format.

Why aren’t unicode separators keys on your keyboard? Maybe that’s suboptimal, but good luck changing that.

Re: Why isn’t there a decent file format for tabular data?

#228

Post by xxs »

Justsignedup wrote: post #15

I literally don't get why JSON is bad: [{row1}, {row2}, {row3}] The fact that it can do more is in no way a negative. Can even make a limited JSON parser with reduced capabilities. And with JSON can do more definitions like header names vs column names vs just arrays of arrays.

That's inferior to even csv in any all its forms. It cannot be appended w/o removing the trailing square bracket. It requires column names per each row, you can't even know how many rows are there w/o parsing. It requires quotes (technical csv allows multiline rows but that's rarely used in practice)

json is absolutely horrid for this type of hack-in jobs. Tabular data just not well structure data, easy to cut and paste.

Re: Why isn’t there a decent file format for tabular data?

#229

Post by pmontra »

zzo38computer wrote: post #221
clarkevans wrote: post #183

Earlier quoted context omitted.

The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.

Do you have the file format documentation?

http://www.dbase.com/Knowledgebase/INT/db7_file_fmt.htm

Re: Why isn’t there a decent file format for tabular data?

#230

Post by pmontra »

zoomablemind wrote: post #180
leeoniya wrote: post #153

Earlier quoted context omitted.

> but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?

What's so special about having a named type for datetime? User will still need to call functions to manipulate the dates. If only for the default display and import?

I've seen a sqlite database with datetimes in three different formats in the same field, because different parts of the application I inherited had different ideas of how to write a datetime and sqlite accepts everything. It's only a string after all.

That's a mistake that the same bad developer couldn't have done with a PostgreSQL or a MySQL.

Post reply on HN