Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

F3

github.com

Post reply on HN
Print view View original

131–140 of 141 posts

Re: F3

#131

Post by JadeNB »

aduffy wrote: post #113
vouwfietsman wrote: post #94

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology wor…

> > Parquet is unfortunately very good just by virtue of being first, and so widely supported

> IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.

I think you and vouwfietsman (https://news.ycombinator.com/item?id=48649412) are actually saying the same thing in different words—I think their "unfortunately" means "it is unfortunate that, by virtue of coming first, this now has a support lead that will make it difficult for anyone else to catch up."

Re: F3

#132

Post by qurren »

vouwfietsman wrote: post #94

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…

> my main gripe with parquet (single table per file) is not even addressed

I consider that simplicity to be a feature, not a shortcoming.

I just tar a bunch of parquets if I need multiple tables. It is beautifully simple and easy to read in any language with its tar and parquet libraries.

Re: F3

#133

Post by jayd16 »

adwn wrote: post #129
jayd16 wrote: post #125

Earlier quoted context omitted.

You still have the halting problem to solve to prevent denial of service.

Solving the halting problem is neither necessary nor sufficient to prevent DoS attacks. It isn't necessary, because settings timeouts or other resource restrictions works way better to prevent DoS. It isn't sufficient, because even if you can prove that a program will halt at some point, this alone doesn't tell you how long it will take. What good does it do to know that the program will run for 10 years before it ha…

It was just a joke, bud. I agree that a timeout would likely be fine depending on the situation. This is basically just a web page with javascript.

Re: F3

#134

Post by ChadNauseam »

anygivnthursday wrote: post #128
titzer wrote: post #86

Earlier quoted context omitted.

But Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.

Yes, if it comes from some reliable partner I can report bugs to, or something built in-house. In such environment probably fine. And maybe thats the main audience and not some open data exchange format where your system may be brought down by someone's random decoder.

But the WASM is integrated into the file. So what scenario would happen where the decode fails? They wrote a file, wrote a wasm decoder, and didn't test the decoder against the file they just wrote? I think it would be a natural step as part of the publish pipeline to run the decoder against the file and compare the output against the source to ensure correctness.

Re: F3

#135

Post by arcfour »

Omega359 wrote: post #75
arcfour wrote: post #24

Earlier quoted context omitted.

Yes...my first thought. No way in hell anyone actually trusts this. (And as if we didn't trust the compiler enough already!)

Meh, it's not that bad. Pretty simple to block inline wasm and to use well known external decoders.

But that is 97% of the value proposition.

Re: F3

#136

Post by anygivnthursday »

ChadNauseam wrote: post #134
anygivnthursday wrote: post #128

Earlier quoted context omitted.

Yes, if it comes from some reliable partner I can report bugs to, or something built in-house. In such environment probably fine. And maybe thats the main audience and not some open data exchange format where your system may be brought down by someone's random decoder.

But the WASM is integrated into the file. So what scenario would happen where the decode fails? They wrote a file, wrote a wasm decoder, and didn't test the decoder against the file they just wrote? I think it would be a natural step as part of the publish pipeline to run the decoder against the file and compare the output against the source to ensure correctness.

Thats what a reliable party would do certainly, which was my point above. But there is no guarantee really. And there may be people testing their decoder on some test data then attaching it to whatever data they send without testing edge cases. Its like testing recovery from backups, people who care do it, many people dont.

Re: F3

#137

Post by apitman »

saulpw wrote: post #124
vouwfietsman wrote: post #94

Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below). Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility , which is (obviously) very hard to improve on, as anything new immediately loses. Parquet is unfortunately very good just by…

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. When I was working with parquet, I imagined a .parquetz file format which was just a zip file containing any number of uncompressed parquet files. So you could sling multiple tables around in a single file, and still use range requests to access them.

tarquet?

Re: F3

#138

Post by ConnorTsui20 »

sanderjd wrote: post #121
aduffy wrote: post #113

Earlier quoted context omitted.

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file. > Parquet is unfortunately very good just by virtue of being first, and so widely supported IMO that is not how technology wor…

Yeah that point about "random access is not the point of columnar formats" fell flat for me for this same reason. Almost since the first day I started using columnar data, I've been interested in solutions that strike this balance between batch and random access. This comes up all the time (in my experience) in data science / ML, where we have use cases for both access patterns against the same data. So I'm with you,…

You should take a look at https://github.com/vortex-data/vortex!

Re: F3

#139

Post by saulpw »

apitman wrote: post #137
saulpw wrote: post #124

Earlier quoted context omitted.

> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up. When I was working with parquet, I imagined a .parquetz file format which was just a zip file containing any number of uncompressed parquet files. So you could sling multiple tables around in a single file, and still use range requests to access them.

tarquet?

Cute name, but alas .tar is not random-access. The .zip format seems to be uniquely simple, ubiquitous, and random-access even for huge files.

Re: F3

#140

Post by sanderjd »

ConnorTsui20 wrote: post #138
sanderjd wrote: post #121

Earlier quoted context omitted.

Yeah that point about "random access is not the point of columnar formats" fell flat for me for this same reason. Almost since the first day I started using columnar data, I've been interested in solutions that strike this balance between batch and random access. This comes up all the time (in my experience) in data science / ML, where we have use cases for both access patterns against the same data. So I'm with you,…

You should take a look at https://github.com/vortex-data/vortex !

Already did after reading your comment :)
Post reply on HN