Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

F3

github.com

Post reply on HN
Print view View original

61–70 of 141 posts

Re: F3

#61

Post by ok123456 »

weinzierl wrote: post #44
jasonjayr wrote: post #13

Earlier quoted context omitted.

So attackers don't have to craft specially corrupted files? They can just include the code to perform the attack in the data file itself?

WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions. If this will pan out security-wise I don't know. I'm more worried that it will be so slow that no one will use it. Interesting idea, though, and I can see applications outside of the "big data" realm this apparently targets.

How do you prevent compression bomb attacks when files can define their own compression functions?

You could have some kind of OOM killer, but that will be a "footgun" that people who are actually doing "big data" will constantly shoot.

This pretty much kills any ingestion pipeline where the source is untrusted.

Re: F3

#62

Post by cbm-vic-20 »

gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

Applets redux.

Re: F3

#64

Post by dj_axl »

largbae wrote: post #3

This could use a bit more "why". Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support... Why should one leave Parquet or ORC for this structure?

Paper mentions Parquet, ORC, Nimble, Lance, TSFile, Bullion, and BtrBlocks.

Re: F3

#65

Post by amluto »

One nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pass over app the data, if that pass doesn’t understand your query, you may have already lost.

I admit I only skimmed the beginning of the paper, and maybe the format is less general than it sounds.

Re: F3

#66

Post by aseipp »

rebeccajae wrote: post #56
gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

It sounds neat, but feels like it might fall apart with higher-complexity formats. What does an embedded decoder for a PDF look like? I guess since they are tightly-coupled to the file bytes themselves, the author of the file gets to choose what formats make sense, but not all formats have a one-true-decode-step.

Despite the name seemingly implying otherwise, F3 is an alternative to columnar storage formats like Parquet; the goal is not to support every conceivable encoding of every file type such as a PDF. Think of the use cases being more like "What if you used a specialized compressor and need a custom block decompression algorithm" or "Decode internal format into Arrow output" or something like that.

Re: F3

#67

Post by hahahacorn »

amluto wrote: post #65

One nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pa…

My understanding is it’s a fallback mechanism

Re: F3

#68

Post by ShinyLeftPad »

To save a click it's a file format for columnar data specifically (like Parquet), which they very generically named Future-proof File Format. Most of this could fit in the title instead of just "F3"

Re: F3

#69

Post by antisthenes »

The description mentions shortcomings of the previous file types like parquet, but it isn't really evident to me what those shortcomings are, or if the use cases for parquet and F3 have really that much of an overlap to make this comparison valid in the first place.

Re: F3

#70

Post by anygivnthursday »

My concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.
Post reply on HN