Viewing profile — ianmcook

I'm one of the core developers of ADBC (Arrow Database Connectivity). This started as a joke — what would it take to call ADBC from COBOL? Turns out, not much. COBOL's fixed-layout…

@1egg0myegg0 that's great to hear. I'll check to see if it applies to Arrow. Another performance issue with DuckDB/Arrow integration that we've been working to solve is that Arrow …

Arrow developer here, we've invested a lot in seamless DuckDB interop, great to see it getting traction. Recent blog post here that breaks down why the Arrow format (which underlie…

Anyone know what format they are serializing the data in to move it between Excel and Python? Are they using Apache Arrow?

Thanks for the heads up. The post is intended to be up but there's an intermittent error happening. It's been reported to the Apache infrastructure team.

Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.

From https://arrow.apache.org/faq/ : "Parquet files cannot be directly operated on but must be decoded in large chunks... Arrow is an in-memory format meant for direct and efficien…

The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a F…

Re this second point: Arrow opens up a great deal of language and framework flexibility for data engineering-type tasks. Pre-Arrow, common kinds of data warehouse ETL tasks like wr…