F3 • Hacker News Board

Re: F3

#101

Post by yung_lean » Tue, Jun 23, 2026, 6:45 PM UTC

grodes wrote: post #82

bluejekyll wrote: post #26

Earlier quoted context omitted.

WASM is platform independent. What do you mean by C bindings? C bindings to what?

C bindings to a C implementation

This isn't using WASM to solve the "how can I make my file format compatible with more programming languages?" problem. This is trying to solve the "how can I add new encodings to my file format without making everyone update their code?" problem. The former would rightly be solved with C bindings that anyone can link with if they want. The latter might not seem like a big deal, but it's been the main blocker advancing the parquet format. Most people end up not caring about new advanced encodings and just write parquet files with the most compatible feature set.

Re: F3

#102

Post by dietr1ch » Tue, Jun 23, 2026, 6:47 PM UTC

largbae wrote: post #3

This could use a bit more "why". Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support... Why should one leave Parquet or ORC for this structure?

I also had no idea what they were talking about, but there's good points about how hardware oblivious and somewhat global is Parquet around metadata.

I found this post interesting,

- https://medium.com/@reliabledataengineering/f3-the-future-pr...

Re: F3

#103

Post by mmaunder » Tue, Jun 23, 2026, 6:49 PM UTC

A Wasm decoder takes encoded bytes and returns an iterator of Arrow Buffers. In case you were wondering.

Re: F3

#104

Post by bguebert » Tue, Jun 23, 2026, 6:50 PM UTC

msla wrote: post #32

verdverm wrote: post #12

Earlier quoted context omitted.

Is embedding executable code into a file a security risk? My assumption is a yes

> Is embedding executable code into a file a security risk? Yes, which is why nobody uses PDFs.

I mean I disable javascript embedded in pdf and feel like it would have been better to not have that feature. It would spare people from the invoice.pdf email attachment viruses because most people had assumed pdf isn't going to be as bad as an exe.

Re: F3

#105

Post by Qerub » Tue, Jun 23, 2026, 6:50 PM UTC

This reminds me of Alan Kay's OOPSLA 1997 presentation "The Computer Revolution Hasn’t Happened Yet" when he describes the Air Force / Burroughs 220 file format from 1961 where the file/tape contained both the data and the procedures to read/write/print them: https://youtu.be/oKg1hTOQXoY?t=1355

Re: F3

#106

Post by mmaunder » Tue, Jun 23, 2026, 6:52 PM UTC

Not quite vaporware, but few commits, PRs, history, actual examples etc. It's pretty thin.

Re: F3

#107

Post by nine_k » Tue, Jun 23, 2026, 6:53 PM UTC

F3 seems to be a reasonable archival data format.

I see many replies criticizing F3 as an operational data format, like Parquet. Of course it can't be made as fast in the general case, or as compatible to the existing infrastructure.

OTOH F3 would be easy to decode into almost any of today's accepted formats, and likely to any of tomorrow's data formats. That's where being self-describing and self-unpacking would be important.

Re: F3

#108

Post by bguebert » Tue, Jun 23, 2026, 6:54 PM UTC

ratorx wrote: post #40

gavinray wrote: post #18

Earlier quoted context omitted.

There is no concept of "executable" vs "non-executable" content in a file. A file is a bag of bytes. You can send those bytes to different things, like a text editor's content-stream, or as the input to a WASM interpreter. What you decide to do with the bytes in a file is your own prerogative. Each byte is whatever you make of it.

There’s a big difference in the expected use of a file. If the file is attacker provided, and the fallback path is being used, the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”. Compare that to JSON. The parser NEVER needs to execute arbitrary instructions. Parser might have bugs, but it avoids a whole class of issues.

I mean json might not be the best example since for a long time people would run json through a javascript engine to parse it but I can see your point.

Re: F3

#109

Post by chatmasta » Tue, Jun 23, 2026, 6:59 PM UTC

As appealing as this is, it will never gain traction without some backwards compatibility with Parquet and wide adoption of query engines to implement that backwards compatible path.

Re: F3

#110

Post by gruntled-worker » Tue, Jun 23, 2026, 7:02 PM UTC

Are we positively sure that WASM will prove to be more future-proof than 640K MS-DOS or WinXP, or SNES cartridge files for that matter? On 6/23/26 there are a lot of emulators that run these. Will WASM necessarily beat them on 6/23/2051? Might be a case of xkcd 927.