Embedding user-defined indexes in Apache Parquet • Hacker News Board

Re: Embedding user-defined indexes in Apache Parquet

#21

Post by ethan_smith » Tue, Jul 15, 2025, 12:53 PM UTC

Nelkins wrote: post #3

Cool, but this is very specific to DataFusion, no? Is there any chance this would be standardized so other Parquet readers could leverage the same technique?

The Arrow/Parquet community is already discussing standardization via the Parquet format GitHub - this approach intentionally uses existing extension points in the format specification to remain compatible while the standardization discussions progress.

Re: Embedding user-defined indexes in Apache Parquet

#22

Post by kernelsanderz » Tue, Jul 15, 2025, 1:53 PM UTC

dkdcio wrote: post #12

dmvinson wrote: post #10

Earlier quoted context omitted.

What are the new file format initiatives you're referencing here? This solution seems clever overall, and finding a way to bolt on features of the latest-and-greatest new hotness without breaking backwards compatibility is a testament to the DataFusion team. Supporting legacy systems is crucial work, even if things need a ground-up rewrite periodically.

Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others https://github.com/lancedb/lance https://github.com/facebookincubator/nimble

I’ve been excited about lancedb and its ability to support vector indexes and efficient row level lookups. I wonder if this approach would work for their design goals and still allow broader backwards compatibility with the parquet ecosystem. Have been intrigued by Ducklake, and they’ve leaned into parquet. Perhaps this approach will allow more flexible indexing approaches with support for the broader parquet ecosystem which is significant.

Re: Embedding user-defined indexes in Apache Parquet

#23

Post by lmeyerov » Tue, Jul 15, 2025, 3:51 PM UTC

jasim wrote: post #17

lmeyerov wrote: post #13

Earlier quoted context omitted.

Yeah I'm happy to see this, we have been curious as part of figuring out cloud native storage extensions to GFQL (graph dataframe-native query lang), and my intuition was parquet was pluggable here... And this is the first I'm seeing a cogent writeup. Likewise, this means, afaict, it's likewise pretty straightforward to do novel indexing schemes within Iceberg as well just by reusing this. The other aspect I've been…

I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644

Thank you, looking forward to reading!

Re: Embedding user-defined indexes in Apache Parquet

#24

Post by hodgesrm » Wed, Jul 16, 2025, 5:08 AM UTC

jasim wrote: post #2

I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…

One question that the article does not cover: compaction. Adding custom indexes means you have to have knowledge of the indexes to compact Parquet files, since you'll want to reindex each time compaction occurs. Otherwise the indexes will at best be discarded. At worst they would even be corrupted.

So it looks as if adopting custom indexes mean you are adopting not just a particular engine for reading but also a particular engine for compaction. That in turn means you can't use generic mechanisms like the compaction mechanism in S3 table buckets. Am I missing something?

Re: Embedding user-defined indexes in Apache Parquet

#25

Post by deepsun » Sat, Jul 19, 2025, 2:51 AM UTC

jasim wrote: post #2

I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…

My main problem with Parquet format is that it depends on Facebook's Thrift (competitor to gRPC).