Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

161–170 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#161

Post by recursive »

mbreese wrote: post #154
recursive wrote: post #120

Earlier quoted context omitted.

If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.

We’re talking about a tabular data file format. If you want to include arbitrary binary, use a binary data file. Or base64 encoded data. Most datasets you’d use data like this for are small enough to fit into memory, so let’s not get carried away. (I happen to use tab delimited files to store data that can’t fit into memory, but that’s okay too)

Yes. I think we're agreeing. I was responding to this "Any binary allowed between the quotes.". Binary data can't generally be dropped directly into a text format without some kind of organized encoding.

Re: Why isn’t there a decent file format for tabular data?

#162

Post by recursive »

tantalor wrote: post #152
recursive wrote: post #120

Earlier quoted context omitted.

If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.

Sure you can, here is example: 0101010010000111101010101

That's just text. In one sense, yes, it's arbitrary bits. But it's also clearly text. It's probably not what was referred to. If you have binary data that's encoded as text, it seems obvious that that can be embedded in text. It probably wouldn't be worth mentioning that "Any binary allowed between the quotes."

Re: Why isn’t there a decent file format for tabular data?

#163

Post by a_t48 »

1vuio0pswjnm7 wrote: post #100
donatj wrote: post #78

> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…

"That's a nightmare to try to edit yourself in a text editor?" I have found these "unusual" separators are useful, e.g., I use ASCII file separator (FS). I use tr and sed to add/remove/change separators. If I had to edit a table interactively I would change the separator to something visible before editing. Thats said, in nvi(1) or less(1), US is displayed as highlighted ^_ and RS as highlighted ^^. It is not difficu…

At that point I’d be considering compiling my own sort. It can’t be that much code.

Re: Why isn’t there a decent file format for tabular data?

#164

Post by gnulinux »

Diggsey wrote: post #106
prepend wrote: post #38

I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…

CSV has more problems than just quoting: one problem is that excel is so commonly used to open CSV files that various weirdnesses in excel have essentially become part of the CSV format. For example, if a value looks like a formula, then the only way to get excel to treat it as text is to put a single quote in front of it, "'=not a formula". This in turn means that its common for values in a CSV to begin with a singl…

I like the format description, thanks. Sounds like something I might implement some day for fun!

The reason I still don't like this is because most of the time I'm sharing csv is to a jira ticket or an email, or to customer and customer insists it needs to be csv.

When customer wants csv there is nothing I can do.

But in other cases I see this CSV file. So it's visual clarity is important. CSV has all the problems you listed, but TSV (and similar, such as your format) has the short coming of relying on an invisible separator. And worse, the width of tab is undefined, undecided.

I don't know what the best trade-of... Maybe we need better tooling so frontends show CSV/TSV et al as separated by cells.

Re: Why isn’t there a decent file format for tabular data?

#167

Post by civilized »

jayd16 wrote: post #145
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

None of them seem all that conducive to source control or merging. Any good format for that?

Why would we expect data to be amenable to version control systems designed for source code?

Re: Why isn’t there a decent file format for tabular data?

#169

Post by ViViDboarder »

neves wrote: post #147
MrPowers wrote: post #114

Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are. Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed ( https://blog.jetbrains…

It's it possible to diff a parquet file?

Is it possible to diff an image?
Post reply on HN