Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn’t there a decent file format for tabular data?

successfulsoftware.net

Post reply on HN
Print view View original

131–140 of 355 posts

Re: Why isn’t there a decent file format for tabular data?

#131

Post by 101011 »

losvedir wrote: post #123
MrPowers wrote: post #114

Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are. Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed ( https://blog.jetbrains…

I've been pretty impressed with parquet lately. One thing I've missed is a way to group tables. Is there a standard for that? While parquet is generally column oriented it has support for metadata about tables of multiple columns. However, I'm not aware of any format that groups the tables , short of just zipping a bunch of files. For context, this would be for an application that passes sqlite files around. So natur…

Is Spark what you're looking for? You can do all sorts of joins, groupings, and aggregations with parquet(s) acting as your source(s)

Re: Why isn’t there a decent file format for tabular data?

#132

Post by _vdpp »

AdamH12113 wrote: post #34

I never understood why the ASCII separator characters aren't used more. It seems like we're one simple text editor feature away from having easy display and modification. Is there some historical reason for not doing that?

I’ve wondered this too. At one point I started writing a hobby programming language and explicitly added escape codes for these extra control characters to be used in string literals. (Ada has them all included in the “ASCII” package which is nice.) It seems like a no-brainer to bake them into file formats like the OP describes.

Re: Why isn’t there a decent file format for tabular data?

#135

Post by jltsiren »

psfried wrote: post #119

> But it is binary, so can’t be viewed or edited with standard tools, which is a pain. I've heard this sentiment expressed multiple times before, and a minor quibble I have with it is that the fact that it's binary has nothing to do with whether or not it's a pain. It's a pain because the tools aren't ubiquitous, so you can't count on them always being installed everywhere. But I'd argue that sqlite _is_ ubiquitous a…

JSON is good for structured data, but I prefer TSV for simple human-readable tabular data. In situations where it's the right choice, a TSV file consists of data and whitespace and nothing else. You can view and edit it with any imaginable tool, and there is no overhead in the form of delimeters and encodings distracting you from the data.

Re: Why isn’t there a decent file format for tabular data?

#136

Post by mastax »

d3mcfadden wrote: post #127
out_of_protocol wrote: post #68

Earlier quoted context omitted.

You can just use sqlite then. Very compact, highly popular (in different role though). Seen it used for large datasets - map tiles (millions of jpeg files). Much smaller size than zip or tar archive, indexed, fast. P.S. sqlite> .mode csv sqlite> .import city.csv cities

Cool, I didn't know about .mode and .import. Super handy tip.

You can also use

    .mode csv
    .headers on
    .output file.csv
Then run a query and it gets output to file.csv.

Re: Why isn’t there a decent file format for tabular data?

#137

Post by earthboundkid »

chubot wrote: post #101

Related thread: CSVs: The Good, the Bad, and the Ugly (2020) https://news.ycombinator.com/item?id=25014721 FWIW I have designed an upgrade to TSV as part of https://www.oilshell.org . Oil actually emits it now from 'pp proc' to pretty print "functions" and their docstrings as a table. It will be used in many parts of Oil, most of which aren't implemented yet. It's called QTT -- Quoted, Typed Tables (formerly QTSV) It…

You should make the connection of QTT and QSN to Rust more clear by coming up with a backronym for RSON.

Re: Why isn’t there a decent file format for tabular data?

#138

Post by prepend »

Diggsey wrote: post #106
prepend wrote: post #38

I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…

CSV has more problems than just quoting: one problem is that excel is so commonly used to open CSV files that various weirdnesses in excel have essentially become part of the CSV format. For example, if a value looks like a formula, then the only way to get excel to treat it as text is to put a single quote in front of it, "'=not a formula". This in turn means that its common for values in a CSV to begin with a singl…

There’s lots of protocols like what you describe and I’ve written a few myself. But they are a pain to teach to other people.

Csv isn’t perfect, but I’d rather live with its weaknesses than use anything else.

Re: Why isn’t there a decent file format for tabular data?

#139

Post by Pxtl »

tony-allan wrote: post #95
Pxtl wrote: post #26

> Why can’t we have a format where Does Excel support it? No? Then that's the end of that. Excel is tabular data to all non developers. The formats supported by Excel are the whole thing. And if we're inventing a CSV-like format that uses a more convenient character than quotes and commas, maybe jumping to a non-displayable non-typeable character isn't the best? Honestly, if I were inventing a table format, I'd use a…

This is a very cool idea but why not just add the closing tags and have a minimal format that is also valid HTML and can easily view the data by pointing your browser at the file? It's already a universal format and as dtech mentions below, you get Excel compatibility. I would also add headers and an optional type attribute... Age If the file is too big then you probably need compress it or use a more appropriate for…

The document I posted was valid html6 -- all of those end tags are optional.

From an XML purist's perspective that's abhorrent, but if we're trying to make a terse table format skipping the close tags is good.

Re: Why isn’t there a decent file format for tabular data?

#140

Post by MobiusHorizons »

thayne wrote: post #118
donatj wrote: post #78

> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) That's a nightmare to try to edit yourself in a text editor? I'd rather just have basically TSV, but with every value always quoted, always UTF-8. Quotes escaped with backslashes, backslashes escaped with backslashes, and that's it. Any binary allowed between the quotes. I deal with CSVs all day every day.…

> That's a nightmare to try to edit yourself in a text editor? You just need a text editor that can support thia format.

So… not a text editor then, right?
Post reply on HN