Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

261–270 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#261

Post by elcritch »

Pxtl wrote: post #139
tony-allan wrote: post #95

Earlier quoted context omitted.

This is a very cool idea but why not just add the closing tags and have a minimal format that is also valid HTML and can easily view the data by pointing your browser at the file? It's already a universal format and as dtech mentions below, you get Excel compatibility. I would also add headers and an optional type attribute... Age If the file is too big then you probably need compress it or use a more appropriate for…

The document I posted was valid html6 -- all of those end tags are optional. From an XML purist's perspective that's abhorrent, but if we're trying to make a terse table format skipping the close tags is good.

SGML originally defined optional tags. The intent was for doc writers to not need to end tags. XML in my view was a 20 year regression..

Re: Why isn’t there a decent file format for tabular data?

#262

Post by _Algernon_ »

thatjoeoverthr wrote: post #254
wodenokoto wrote: post #215

Earlier quoted context omitted.

But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.

It’s a very meta-level problem. The main problem with CSV is “this is CSV” is an ambiguous statement. One could commit to a rule set, and give it a new name. Probably, someone has done this already.

Sounds like xkcd 927

Re: Why isn’t there a decent file format for tabular data?

#263

Post by qwerty456127 »

Gigachad wrote: post #86
quesera wrote: post #36

Earlier quoted context omitted.

ASCII separators (mnemonics FS, GS, RS, US) are difficult for most users to type, and have no obvious/standardized visual representation.

Who is typing out CSVs anyway. You’d either use a spreadsheet tool or do it in code.

From my point of view, CSV is mostly operated in text editors because they are much faster and don't risk to do any unintended modifications.

Re: Why isn’t there a decent file format for tabular data?

#264

Post by hoseja »

eru wrote: post #249
hoseja wrote: post #247

Why are all these "separated"? Sounds like null-terminated C strings. Wouldn't it be better to put the dimension in a header and then just dump the contents unseparated? Why the need to view them in simple text editors?

What you suggested works for fixed width fields only, doesn't it?

Yeah, true. Still, one or more bytes of overhead per cell and row/column seems wasteful.

Re: Why isn’t there a decent file format for tabular data?

#265

Post by larsrc »

jarofgreen wrote: post #244

Fundamentally, why are people so concerned about about hand editing tabular data? The options for that are pretty terrible. Only editing an existing value in one cell is ok. Everything else - adding new rows, new columns and making sure the file format and column structure is correct - is all terrible. This gets doubly so if you stick it in git and start getting messy pull requests. I would carefully question what us…

I regularly need to hand-edit tabular data I have generated from other data when it turns out there are a few weird cases I didn't handle right. I do a lot of one-off data processing in several stages, so it's not just a question of fixing the generator. If I want to edit the structure, sure, I'll take it into Sheets, but for data that's slightly weird, a text editor or shell pipe is often faster.

Re: Why isn’t there a decent file format for tabular data?

#266

Post by vram22 »

clarkevans wrote: post #183
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.

>The DBT (DBase III) format was common in the 80s and 90s.

That should actually be DBF, the format for the main database tables. DBT was an ancillary format for the memo fields, which were used to store longer pieces of text in one column of a DBF record. Overall, that generic format and associated software is called XBASE. People still use it in production. And data entry (CRUD) using it with plain text or TUI DOS-style apps blows Web and even GUI apps out of the water in speed of use.

Re: Why isn’t there a decent file format for tabular data?

#268

Post by srik »

Brometheus wrote: post #255
lordnacho wrote: post #233

Earlier quoted context omitted.

Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.

ASCII Code 29: Group Separator ASCII Code 30: Record Separator ASCII Code 31: Unit Separator https://theasciicode.com.ar/ascii-control-characters/record-...

Thank you for pointing them out. I've browsed, and searched through that ascii table countless times, usually looking for something in particular, yet never realized that there were such a thing as separators before.

Re: Why isn’t there a decent file format for tabular data?

#269

Post by AdrenalinMd »

jayd16 wrote: post #145
asperous wrote: post #126

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…

None of them seem all that conducive to source control or merging. Any good format for that?

There is a git diff driver for CSV that makes the CSV diffing and merging a lot better. https://paulfitz.github.io/2014/07/09/diff-merge-csv.html

Works like a charm!

Re: Why isn’t there a decent file format for tabular data?

#270

Post by derriz »

MrPowers wrote: post #114

Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are. Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed ( https://blog.jetbrains…

Parquet is still relatively poorly supported in the JVM world unless this changed in the last year? Yes, you can use Spark but that's an absolutely huge dependency just to read a file representing a table of data. The alternative - trying to use poorly documented Hadoop libraries - was only marginally better. Maybe the story has changed in the last year?

The other problem with Parquet is that it's overly flexible/supports application-specific metadata. It's all fine when you use a single tool/library for reading and writing files but cross-platform is problematic. Saving a Pandas dataframe to parquet, for example, will include a bunch of Pandas-specific metadata which is ignored or skipped by other libraries.

In this case, it required converting timestamp/datetime columns to a nano time int64 representation before writing data from Pandas (for example), otherwise you could not read those columns using anything that wasn't Pandas.

But maybe this has changed but at the time I last used parquet as a table format?

Post reply on HN