111–120 of 141 posts
Are we positively sure that WASM will prove to be more future-proof than 640K MS-DOS or WinXP, or SNES cartridge files for that matter? On 6/23/26 there are a lot of emulators that run these. Will WASM necessarily beat them on 6/23/2051? Might be a case of xkcd 927.
Are we positively sure that WASM will prove to be more future-proof than 640K MS-DOS or WinXP, or SNES cartridge files for that matter? On 6/23/26 there are a lot of emulators that run these. Will WASM necessarily beat them on 6/23/2051? Might be a case of xkcd 927.
I really enjoy the spirit that moved you to write this comment. I want my AI Agents to adopt this attitude.
BTW, while we're on the topic. I don't do social media. I occasionally type up a text or post on a technical board. Maybe 98% of my textual interaction these days is with LLMs. I would not be surprised if my prose changes to resemble theirs over time. I suppose that's symbiosis for ya. It's possible that your AI-dar might get even more ineffective.
Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…
This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file.
> Parquet is unfortunately very good just by virtue of being first, and so widely supported
IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.
> Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics
Fast analytics, as well as newer ML-shaped workloads, are inherently mix of batch scans and random access.
Some of the authors of F3 previously authored another paper that goes into the details of the shortcomings of Parquet
https://www.vldb.org/pvldb/vol17/p148-zeng.pdf
All of the newer formats that popped up recently (Vortex, Lance, F3 now) have been working on solving the problems outlined in that paper.
Lance has some interesting ideas, Vortex focuses on extensibility and performance by replacing all of Parquet's black-box encoders with fully transparent encodings. This solves the tradeoff between bulk and element decoding, allowing you to have efficient full scans and really fast random access.
E.g. Langchain recently rebuilt a system that used to be all Parquet files to use Vortex and saw a massive speedup, which they talk about more here: https://www.langchain.com/blog/introducing-smithdb
Disclaimer: I work on Vortex, so a lot of these questions about "what is the point of building a new format" are things that I have grappled with myself.
F3: Open-source data file format for the future [pdf] - https://news.ycombinator.com/item?id=45437759 - Oct 2025 (125 comments)
plus this bit:
An Open File Format for storing the information from a forge - https://news.ycombinator.com/item?id=44043253 - May 2025 (1 comment)
Don't know why but I had to think of https://xkcd.com/2116/
Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…
> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology wor…
Sure it would, you can attach a multi-table sqlite database in duckdb
> that does not mean just because it came first
I agree with most of your points, I am not stating my opinion but my observations. I am the target audience here, I want to use this, but I don't really care too much about the file format itself, at least not as much as I care about the data inside.
That means access, which means compatibility with my tooling.
Compatibility is hard to beat.
This is the concorde of file formats.
If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet. Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.
the same sandboxing capability exists for WASM as well.
it is actually better for long-term archival: you dont need to carry decompression program, since it will be a part of the archive file itself
Earlier quoted context omitted.
> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology wor…
> DuckDB wouldn't really know what to do with a Sure it would, you can attach a multi-table sqlite database in duckdb > that does not mean just because it came first I agree with most of your points, I am not stating my opinion but my observations. I am the target audience here, I want to use this, but I don't really care too much about the file format itself, at least not as much as I care about the data inside. Tha…
FWIW I think if you are just doing pure analytics and nothing else, Parquet will probably continue to do the job for you just fine, and you don't need to touch your workloads at all.
These new formats I think will find a niche where people aren't just running Spark jobs, but doing lots of systems building over large tables. If you're building a PB-scale data warehouse, you care a lot about the file format b/c it is a big factor in your performance curve, and you're willing to ship new experimental codecs in response to new datatypes you want to support that the system wasn't originally designed for, or you want to use a newly invented compressor.
Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…
Frankly it's a change from the usual ChatGPT generated slop that most landing pages are these days.
F3 seems to be a reasonable archival data format. I see many replies criticizing F3 as an operational data format, like Parquet. Of course it can't be made as fast in the general case, or as compatible to the existing infrastructure. OTOH F3 would be easy to decode into almost any of today's accepted formats, and likely to any of tomorrow's data formats. That's where being self-describing and self-unpacking would be…