Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

F3

github.com

Post reply on HN
Print view View original

51–60 of 141 posts

Re: F3

#51

Post by ratorx »

gavinray wrote: post #41
ratorx wrote: post #40

Earlier quoted context omitted.

There’s a big difference in the expected use of a file. If the file is attacker provided, and the fallback path is being used, the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”. Compare that to JSON. The parser NEVER needs to execute arbitrary instructions. Parser might have bugs, but it avoids a whole class of issues.

> the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”. And then do what with it? WASM physically cannot interact with the underlying host or perform I/O -- you need a WASI environment for that.

Putting aside the WASM sandboxing (I’m not familiar enough with it to understand how sandboxing works) there’s a DoS vector at least. Even regexes have had many DoS issues, and I can’t imagine WASM being easier to sandbox for DoS risk.

Re: F3

#52

Post by yung_lean »

andrewstuart2 wrote: post #23
gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

I would call it clever. I'm not sure I'd call it genius. When I'm working with data I'm working in a specific set of languages. Usually one. Yeah, other people might be working in other languages, but no individual author really needs a language-agnostic way of accessing data beyond compile time. Add to that the likely runtime boundaries that may need to be crossed instead of e.g. inlined by the compiler because it's…

A big problem with parquet, which this aims to replace, is that it's hard to add new encodings because everyone wants to stay compatible with old readers. Embedding the decoders in the file as WASM solves this problem since in theory, old readers will be able to read new files by just using the provided WASM to decode a column whose format the reader doesn't recognize.

So this is really about making a file that is forwards compatible in a way that lets you push the standards more than existing formats.

Re: F3

#53

Post by coffeecoders »

If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet.

Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.

Re: F3

#54

Post by bilekas »

weinzierl wrote: post #44
jasonjayr wrote: post #13

Earlier quoted context omitted.

So attackers don't have to craft specially corrupted files? They can just include the code to perform the attack in the data file itself?

WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions. If this will pan out security-wise I don't know. I'm more worried that it will be so slow that no one will use it. Interesting idea, though, and I can see applications outside of the "big data" realm this apparently targets.

> The decoders don't need a lot of access, they can basically be pure functions

They don't currently either do they? It's the tight coupling of the interface layer no? I'm not sure this would be faster, or more secure so reliability might be the best usecase?

Re: F3

#56

Post by rebeccajae »

gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

It sounds neat, but feels like it might fall apart with higher-complexity formats. What does an embedded decoder for a PDF look like? I guess since they are tightly-coupled to the file bytes themselves, the author of the file gets to choose what formats make sense, but not all formats have a one-true-decode-step.

Re: F3

#57

Post by 0xbadcafebee »

coffeecoders wrote: post #53

If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet. Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.

You don't want to run a custom 10-year-old data parsing function every time you read a single data record?

Re: F3

#58

Post by 7373737373 »

ratorx wrote: post #51
gavinray wrote: post #41

Earlier quoted context omitted.

> the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”. And then do what with it? WASM physically cannot interact with the underlying host or perform I/O -- you need a WASI environment for that.

Putting aside the WASM sandboxing (I’m not familiar enough with it to understand how sandboxing works) there’s a DoS vector at least. Even regexes have had many DoS issues, and I can’t imagine WASM being easier to sandbox for DoS risk.

There exist Wasm interpreters capable of limiting the number of instructions executed.

Re: F3

#59

Post by jedberg »

gavinray wrote: post #37
jedberg wrote: post #28

Earlier quoted context omitted.

Sure, but when the standard says "read this file and execute the instructions you find at the beginning" that is more dangerous than "this is a file with data and your program needs to figure out how to read it".

I guess it's a good thing that the F3 standard does not say "read this file and execute the instructions you find at the beginning", then? The WASM encoders/decoders are embedded resources that exist as byte offsets in the file metadata, not header info.

Ok if you want to be pedantic, the standard says, "if you can't read this file, go to the offset and then execute the code you find" which isn't functionally different from what I said.

Re: F3

#60

Post by zerobees »

Some folks described it as genius. I guess it's my turn to play the role of an annoying HN skeptic: I find it somewhat silly. Data compression formats are secondary to what you're planning to do with the data once decoded. An audio file is completely different than an SVG image. An embedded VM that decompresses video to raw pixels doesn't magically let you play that video in a text editor, so there's no radically new kind of interoperability. Each new format still needs to be handled in a format-specific way.

I guess one use case is that I come up with a video compression scheme that's better than H.265, but not all platforms support it, so I embed a decoder that would allow me to play it back on legacy hardware. But that also shows the weakness of the idea: it's unlikely that legacy hardware will perform well doing software-only decode for video formats from the future. If we rolled this idea out in the 1990s, it would not have allowed watching Netflix on an i386.

In the same vein, I doubt this would have allowed me to open Word 2021 files in Word 97. There's no 1-to-1 mapping between the data structures. So if this kind of compat isn't slam-dunk, what's the goal?

The downsides are clear. First, it's probably a maintenance nightmare: if your decoder has a bug that needs fixing, how do you patch all the files that already embed it? And then, there's size overhead and security risks. We're adding a considerable attack surface to every format parser. It's more opportunities for remote code execution, resource exhaustion attacks, and so on. Again, this is not always wrong, but what's the benefit?

Post reply on HN