Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

F3

github.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: F3

#121

Post by sanderjd »

aduffy wrote: post #113
vouwfietsman wrote: post #94

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology wor…

Yeah that point about "random access is not the point of columnar formats" fell flat for me for this same reason. Almost since the first day I started using columnar data, I've been interested in solutions that strike this balance between batch and random access. This comes up all the time (in my experience) in data science / ML, where we have use cases for both access patterns against the same data.

So I'm with you, I'm very unconvinced that parquet (and the various things that are parquet or essentially-parquet under the hood) are the end of the line here.

Re: F3

#122

Post by alex7o »

So you know what is a file format the we would be able to Reed 100 years from now. CSV, json even fits (that is 30 years old now). If you don't know the original way it was created you know what each field I supposed to mean (if done well). Otherwise you look at hex decoded data with no way if knowing how to decide it if you don't have tha spec on how and why this was encoded. Msgpack and cbor are cool but in 100 years there is no way to decide it.

Re: F3

#124

Post by saulpw »

vouwfietsman wrote: post #94

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.

When I was working with parquet, I imagined a .parquetz file format which was just a zip file containing any number of uncompressed parquet files. So you could sling multiple tables around in a single file, and still use range requests to access them.

Re: F3

#125

Post by jayd16 »

mirashii wrote: post #17
verdverm wrote: post #12

Earlier quoted context omitted.

Is embedding executable code into a file a security risk? My assumption is a yes

That would be why it chose a VM that is explicitly designed for sandboxing rather than native executable code or similar, the risk can be minimized by reducing the surface area available to that executable code to almost nothing.

You still have the halting problem to solve to prevent denial of service.

Re: F3

#128

Post by anygivnthursday »

titzer wrote: post #86
anygivnthursday wrote: post #70

My concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.

But Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.

Yes, if it comes from some reliable partner I can report bugs to, or something built in-house. In such environment probably fine. And maybe thats the main audience and not some open data exchange format where your system may be brought down by someone's random decoder.

Re: F3

#129

Post by adwn »

jayd16 wrote: post #125
mirashii wrote: post #17

Earlier quoted context omitted.

That would be why it chose a VM that is explicitly designed for sandboxing rather than native executable code or similar, the risk can be minimized by reducing the surface area available to that executable code to almost nothing.

You still have the halting problem to solve to prevent denial of service.

Solving the halting problem is neither necessary nor sufficient to prevent DoS attacks.

It isn't necessary, because settings timeouts or other resource restrictions works way better to prevent DoS.

It isn't sufficient, because even if you can prove that a program will halt at some point, this alone doesn't tell you how long it will take. What good does it do to know that the program will run for 10 years before it halts? By that time, service will already have been denied. Even turning hash table lookups from O(1) to O(n) (still very much terminating!) can result in a DoS.

Post reply on HN