Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

F3

github.com

Post reply on HN
Print view View original

91–100 of 141 posts

Re: F3

#91

Post by blmarket »

johncolanduoni wrote: post #73
ok123456 wrote: post #61

Earlier quoted context omitted.

How do you prevent compression bomb attacks when files can define their own compression functions? You could have some kind of OOM killer, but that will be a "footgun" that people who are actually doing "big data" will constantly shoot. This pretty much kills any ingestion pipeline where the source is untrusted.

OOM killing in WebAssembly is trivial, since it’s all in a growable linear memory. All the runtimes I’m aware of have a simple maximum memory setting, and they’ll trap any allocation requests after that point.

Attack is not just on file format itself. Based on the function signature it's possible for a single decoder to generate infinite bytestream - makes a lot of headache to reader implementation - implementing STRLEN is no longer trivial question.

Either engines should put some limit (e.g. VARCHAR(2000) to enforce length to be limited to 2000, but there are some other engines supporting unlimited BLOBs), or decoder should give a hint what is the maximum length it will yield. Unfortunately current research level project does not have such considerations implemented yet...

Re: F3

#92

Post by titzer »

Retr0id wrote: post #83
titzer wrote: post #79

Earlier quoted context omitted.

That exploit targeted an integer overflow in a bespoke Apple sandboxing mechanism. Bespoke sandboxing mechanisms have weird bugs. Not that Wasm engines don't have bugs, but the whole point is to have an extremely solid, well-specified and efficient implementation of a widely accepted bytecode format. We can scope down the capabilities given to any program to a minimal set.

Bugs are near-inevitable, and mitigations are the last line of defence. Scripting engines are excellent for bypassing mitigations (iiuc in the case of the FORCEDENTRY exploit, it was used for adjusting ASLR'd offsets). As a random example that's an area of personal interest to me, I know of 3 distinct methods of achieving userland ROP execution of the Nintendo Switch 2, and all three rely on the (ab)use of a scriptin…

Well don't accept code from anyone ever then.

But seriously, if your format requires extensibility to the point that it embeds a bytecode, especially a Turing-complete bytecode, what format are you going to choose? Just design a new one? That's how you end up with a scripting engine with three ROP exploits.

Re: F3

#93

Post by ok123456 »

computomatic wrote: post #85
ok123456 wrote: post #61

Earlier quoted context omitted.

How do you prevent compression bomb attacks when files can define their own compression functions? You could have some kind of OOM killer, but that will be a "footgun" that people who are actually doing "big data" will constantly shoot. This pretty much kills any ingestion pipeline where the source is untrusted.

It seems like the WASM is simply a fallback if no other decoder is available. If the data source is untrusted, simply don’t run the WASM decoders. “Some code is untrusted” does not mean code should never be executed. There are more use cases with trusted sources than untrusted.

So I define the data type to be "asdklfjaslkdfjiolsadfjoiusadfoiasfoikasjfdoisadf" and give you a decoder for it.

Re: F3

#94

Post by vouwfietsman »

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below).

Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility, which is (obviously) very hard to improve on, as anything new immediately loses.

Parquet is unfortunately very good just by virtue of being first, and so widely supported. The most widely used parquet version is the oldest version from 2013 (as per the paper itself), so parquet itself couldn't even supplant parquet. If you want to improve on it, you need to bring some serious results, which I don't think f3 does.

Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.

Also also, it seems to go out of its own way to include a compiled wasm binary for decoding, yet requires flatbuffers to parse that blob? Kind of defeats the purpose.

Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics. F3 seems to sacrifice fast analytics for the wasm decoder. I don't get it.

Maybe I'm being too cynical. Can someone help me out here?

https://dl.acm.org/doi/epdf/10.1145/3749163

Re: F3

#95

Post by coldtea »

grodes wrote: post #16
gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

How is wasm better than C bindings?

C bindings are not platform independent, nor do they come with a runtime and a sandbox, among other things. Apples to oranges.

Re: F3

#96

Post by NooneAtAll3 »

msla wrote: post #32
verdverm wrote: post #12

Earlier quoted context omitted.

Is embedding executable code into a file a security risk? My assumption is a yes

> Is embedding executable code into a file a security risk? Yes, which is why nobody uses PDFs.

which is why no sane pdf viewer implements executable features*

Re: F3

#97

Post by vouwfietsman »

gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

except you need flatbuffers to access that blob

Re: F3

#98

Post by coldtea »

andrewstuart2 wrote: post #23
gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

I would call it clever. I'm not sure I'd call it genius. When I'm working with data I'm working in a specific set of languages. Usually one. Yeah, other people might be working in other languages, but no individual author really needs a language-agnostic way of accessing data beyond compile time. Add to that the likely runtime boundaries that may need to be crossed instead of e.g. inlined by the compiler because it's…

>no individual author really needs a language-agnostic way of accessing data beyond compile time.

That's so untrue! People need language-agnostic ways to access data all the time, and people work with data accessing them from multiple languages all the time!

If I have parquet files I can load them in duckdb, in pandas and polars, process them with various independent tools, and loads of other things... and people do that.

This is also why people like something like an SQL database, your data is not locked to some specific language / lib for access.

Re: F3

#100

Post by gavinray »

mort96 wrote: post #81
gavinray wrote: post #9

This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist: > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native…

I don't understand how that's supposed to work. What does the decoder decode into ? That's gonna depend entirely on the kind of data, right? For some formats, it's gonna be a stream of bytes; for others, a 2D plane of pixels; others again will need vertexes, 2D planes of pixels and UV maps; for some, an object graph will make more sense.

It appears as though the WASM decode returns two values -- one indicating the data type as a primitive value, and a second value being the data buffer

Then there is a helper in this case to de-serialize, "primitive_array_from_buffers()"

https://github.com/future-file-format/F3/blob/bd92506447dc13...

Post reply on HN