Viewing profile — wesm

See also https://wesmckinney.com/blog/mythical-agent-month/

See also The Mythical Agent-Month https://wesmckinney.com/blog/mythical-agent-month/

Super well! I don't work without this tool running in the background supervising all the agents' work

I've been building https://roborev.io/ (continuous background code review for agents) essentially as a cope to supervise the poor quality of the agents' work, since my agents write…

I was especially excited to learn that RZ is built on Apache Arrow internally, which makes it easy to integrate with other Arrow-based applications and the emerging "Composable Dat…

If you read my slide decks over the last 7 years or so (while I've been working actively on Arrow and sibling projects like Ibis) I've been saying exactly this. See e.g. https://ib…

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhou…

I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data st…

> Arrow's serialization is Protobuf Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.

There's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute direc…

Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engine…

(Wes here) I appreciate the Arrow shout-out but note that Apache Arrow has been a major open source community collaboration and not something I can take sole credit for.

There is no “JIRA politics” blocking the LZ4 work, only a lack of volunteers to do the development and testing.

Keep in mind that Arrow Java C++/Python interop has been in production use in Apache Spark and elsewhere for multiple years now. We have avoided some of the mistakes of past projec…

Again, I have to object to your use of “arrow/parquet”. These are not the same open source projects and while people use them together it isn’t fair to the developers of each proje…

See http://arrow.apache.org/faq/index.html#what-about-arrow-file... You can store them long-term if you want (and you'll still be able to read them 5 years from now) but we aren't …

What you've written sounds like a criticism of the JVM data analytics ecosystem (the Java Parquet library in particular) and not Apache Arrow itself. Parquet for Java is an indepen…