Note that there are "Puffin files" associated with Iceberg which have some overlap with this functionality: https://iceberg.apache.org/puffin-spec/#file-structure
Embedding user-defined indexes in Apache Parquet
11–20 of 25 posts
Re: Embedding user-defined indexes in Apache Parquet
#12I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…
What are the new file format initiatives you're referencing here? This solution seems clever overall, and finding a way to bolt on features of the latest-and-greatest new hotness without breaking backwards compatibility is a testament to the DataFusion team. Supporting legacy systems is crucial work, even if things need a ground-up rewrite periodically.
Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others
Re: Embedding user-defined indexes in Apache Parquet
#13I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…
Yeah I'm happy to see this, we have been curious as part of figuring out cloud native storage extensions to GFQL (graph dataframe-native query lang), and my intuition was parquet was pluggable here... And this is the first I'm seeing a cogent writeup.
Likewise, this means, afaict, it's likewise pretty straightforward to do novel indexing schemes within Iceberg as well just by reusing this.
The other aspect I've been curious about is the happy path pluggable types for custom columns. This shows one way, but I'm unclear if same thing.
Re: Embedding user-defined indexes in Apache Parquet
#14I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…
What are the new file format initiatives you're referencing here? This solution seems clever overall, and finding a way to bolt on features of the latest-and-greatest new hotness without breaking backwards compatibility is a testament to the DataFusion team. Supporting legacy systems is crucial work, even if things need a ground-up rewrite periodically.
Off the top of my head:
- Vortex https://github.com/vortex-data/vortex
- Lance https://github.com/lancedb/lance
- Nimble https://github.com/facebookincubator/nimble
There are also a bunch of ideas coming out of academia, but I don't know how many of them have a sustained effort behind them and not just a couple of papers
Re: Embedding user-defined indexes in Apache Parquet
#15Earlier quoted context omitted.
Parquet files include a field called key_value_metadata in the FileMetadata structure; it sits in the footer of the file. See: https://github.com/apache/parquet-format/blob/master/src/mai... The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parqu…
So, can we take that as a "no"?
There is no spec. Personally I hope that the existing indexes (bloom filters, zone maps) get re-designed to fit into a paradigm where parquet itself has more first class support for multiple levels of indexes embedded in the file and conventions for how those common types. That is, start with Wild West and define specs as needed
Re: Embedding user-defined indexes in Apache Parquet
#16Note that there are "Puffin files" associated with Iceberg which have some overlap with this functionality: https://iceberg.apache.org/puffin-spec/#file-structure
Speaking of Puffin files, Apache Parquet always makes me think of this 1978 SNL intro with Bill Murray and SNL bass player Buddy Williams:
Re: Embedding user-defined indexes in Apache Parquet
#17I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…
Yeah I'm happy to see this, we have been curious as part of figuring out cloud native storage extensions to GFQL (graph dataframe-native query lang), and my intuition was parquet was pluggable here... And this is the first I'm seeing a cogent writeup. Likewise, this means, afaict, it's likewise pretty straightforward to do novel indexing schemes within Iceberg as well just by reusing this. The other aspect I've been…
I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644
Re: Embedding user-defined indexes in Apache Parquet
#18I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age. One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.) This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility wit…
Yeah I'm happy to see this, we have been curious as part of figuring out cloud native storage extensions to GFQL (graph dataframe-native query lang), and my intuition was parquet was pluggable here... And this is the first I'm seeing a cogent writeup. Likewise, this means, afaict, it's likewise pretty straightforward to do novel indexing schemes within Iceberg as well just by reusing this. The other aspect I've been…
We are actively working on supporting extension types. The mechanism is likely to be using the Arrow extension type mechanism (a logical annotation on top of existing Arrow types https://arrow.apache.org/docs/format/Columnar.html#format-me...)
I expect this to be used to support Variant https://github.com/apache/datafusion/issues/16116 and geometry types
(note I am an author)
Re: Embedding user-defined indexes in Apache Parquet
#19Earlier quoted context omitted.
So, can we take that as a "no"?
There is no spec. Personally I hope that the existing indexes (bloom filters, zone maps) get re-designed to fit into a paradigm where parquet itself has more first class support for multiple levels of indexes embedded in the file and conventions for how those common types. That is, start with Wild West and define specs as needed
> That is, start with Wild West and define specs as needed
Yes this is my personal hope as well -- if there are new index types that are widespread, they can be incorporated formally into the spec
However, changing the spec is a non trivial process and requires significant consensus and engineering
Thus the methods used in the blog can be used to use indexes prior to any spec change and potentially as a way to prototype / prove out new potential indexes
(note I am an author)
Re: Embedding user-defined indexes in Apache Parquet
#20Earlier quoted context omitted.
But does data fusion publish a specification of how this metadata can be read, along with a test suite for verifying implementations? Because if they don't, this cannot be reliably used by any other impl
Parquet files include a field called key_value_metadata in the FileMetadata structure; it sits in the footer of the file. See: https://github.com/apache/parquet-format/blob/master/src/mai... The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parqu…
> Note that the readers of Parquet need to be aware of any metadata to exploit it. But if not, nothing changes
The one downside of this approach, which is likely obvious, but I haven't seen mentioned is that the resulting parquet files are larger than they would be otherwise, and the increased size only benefits engines that know how to interpret the new index
(I am an author)