Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

181–190 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#181

I mean, in business contexts at least, Excel is what we end up using nearly all the time because it's ubiquitous. This doesn't mean there aren't issues with it; obviously it's nonFree, and it's notorious for making assumptions about data (see recent coverage of scientists literally renaming a gene (I think?) because Excel kept assuming it was a date or something). But Excel as a tool is on so many desktops that you c…

[deleted]

Re: Why isn’t there a decent file format for tabular data?

#182

Earlier quoted context omitted.

You are limited to the basic types (int, floating point, string, and blob), however. I can somewhat get behind the opinionated argument for not needing more specific types like most common language types, but not the lack of a date type.

> but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?

In the context of this thread–a decent file format for tabular data–SQLite's type system peculiarities are irrelevant.

Re: Why isn’t there a decent file format for tabular data?

#183

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.

Re: Why isn’t there a decent file format for tabular data?

#187
post #145

Earlier quoted context omitted.

None of them seem all that conducive to source control or merging. Any good format for that?

CSV should be pretty easy to merge, given that it’s line based.

We use Subversion to manage CSV files pretty much constantly.

Re: Why isn’t there a decent file format for tabular data?

#188

Earlier quoted context omitted.

Not being able to include Tabs and Carriage Returns in your data can be a problem though.

Unicode just needs a single special delimiter character that is only used as a delimiter.

Ideally a separate row and a column delimiter

Re: Why isn’t there a decent file format for tabular data?

#189
post #102

Earlier quoted context omitted.

What's the difference between quoted and escaped delimiters? (Keeping in mind that escaping sequences can themselves be escaped, ad infinitum. You can't simply seek to an escape sequence and depend algorithmically on a small, fixed lookbehind.)

I think the parent that if newlines were encoded as "\n" (with a backslash) then you could always split on (actual) newlines and process them in parallel without having to tokenize the quote first.

Yep that’s exactly it.

Re: Why isn’t there a decent file format for tabular data?

#190

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

The author doesn't like any of those tradeoffs and wishes to make another one, what's the problem with that ?

You don't believe the design space is exhaustively explored by the designs and protocols you mentioned, do you? there is always another local optimum to be found.

Post reply on HN