A fast path for decoding fixed-length lists in Parquet

#1

Post by fanf2 » Wed, Jul 29, 2026, 8:42 PM UTC

A fast path for decoding fixed-length lists in Parquet
morling.dev

Re: A fast path for decoding fixed-length lists in Parquet

#2

Post by petrizhang » Wed, Aug 05, 2026, 5:59 AM UTC

I ran into exactly this while profiling a Parquet-based vector index on DataFusion. The vectors all have a fixed dimension, but LIST decoding and rebuilding the Arrow arrays still showed up as a significant CPU cost. Have you looked at whether a similar fast path could be added to parquet-rs?

Re: A fast path for decoding fixed-length lists in Parquet

#3

Post by gunnarmorling » Wed, Aug 05, 2026, 9:16 PM UTC

petrizhang wrote: post #2

I ran into exactly this while profiling a Parquet-based vector index on DataFusion. The vectors all have a fixed dimension, but LIST decoding and rebuilding the Arrow arrays still showed up as a significant CPU cost. Have you looked at whether a similar fast path could be added to parquet-rs?

I haven't looked at parquet-rs specifically, but I can't think of any reason why it couldn't implement this kind of optimization.

Re: A fast path for decoding fixed-length lists in Parquet

#4

Post by petrizhang » Sat, Aug 08, 2026, 4:08 AM UTC

gunnarmorling wrote: post #3

petrizhang wrote: post #2

I ran into exactly this while profiling a Parquet-based vector index on DataFusion. The vectors all have a fixed dimension, but LIST decoding and rebuilding the Arrow arrays still showed up as a significant CPU cost. Have you looked at whether a similar fast path could be added to parquet-rs?

I haven't looked at parquet-rs specifically, but I can't think of any reason why it couldn't implement this kind of optimization.

Thanks! I’ll take a closer look at parquet-rs and see if I can prototype a fast path for fixed-size, non-null lists.