Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sparrow, a modern C++ implementation of the Apache Arrow columnar format

johan-mabille.medium.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#21

Post by tempodox »

quietbritishjim wrote: post #10
senkora wrote: post #6

Earlier quoted context omitted.

This bothered me enough to check the source code, because I simply had to know: template std::pair get_arrow_structures(A& a) { arrow_proxy& proxy = detail::array_access::get_arrow_proxy(a); return std::make_pair(&(proxy.array()), &(proxy.schema())); } https://github.com/man-group/sparrow/blob/c01a768f590ebf3b22... So the answer is that the `std::move` does nothing and should be omitted, because this function only ha…

> So the answer is that the `std::move` does nothing and should be omitted You can't assign an rvalue into an lvalue reference, precisely to avoid this sort of mistake. If this is the only overload then this just wouldn't compile (e.g. [1]). So the std::move isn't doing nothing, but yes it should be omitted. Maybe it's just a weird typo. [1] https://ideone.com/4NS5dI

Indeed, I don't even have to compile it, vim (+ Syntastic) tells me as soon as it sees it:

  candidate function not viable: expects an lvalue for 1st argument

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#22

Post by remexre »

jandrewrogers wrote: post #5
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Not speaking to the specific design choices here, but in C++ moved-from objects are not destroyed and must be valid in their moved-from state (e.g. a sentinel value to indicate they’ve been moved) so that they can be destroyed in the indefinite future. This is useful even though “destroy on move” is the correct semantics for most cases. Making “move” and “destroy” distinct operations increases the flexibility and exp…

If an object is a RAII container though, shouldn't moving transfer the ownership of the contents to the destination? Otherwise there's no way to increase the lifetime of the contents.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#24

Post by amluto »

jandrewrogers wrote: post #5
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Not speaking to the specific design choices here, but in C++ moved-from objects are not destroyed and must be valid in their moved-from state (e.g. a sentinel value to indicate they’ve been moved) so that they can be destroyed in the indefinite future. This is useful even though “destroy on move” is the correct semantics for most cases. Making “move” and “destroy” distinct operations increases the flexibility and exp…

> A common case where this is useful is if the address space where the object lives is accessible, for read or write, by references exogenous to the process like some kinds of shared memory or hardware DMA.

Huh? Okay, I allocate some memory and DMA-map it to the device [0]. Then:

1. Device uses it.

2. I copy or otherwise consume the data. Hopefully not in a way that causes UB.

3. I’m logically done, but maybe the device isn’t.

4. I’m 100% done, and I unmap the memory. And it’s an error if the device touches it again.

Why would I represent steps 2 and 3 as std::move? Maybe this results in efficient code in some particular code base, but it is not idiomatic, and I can almost guarantee that the same performance could be achieved in a less mind-bending and expectation-defying manner that doesn’t call itself a move.

I don’t see why Rust would have any particular trouble with this as long as you don’t try to force the lifetime of a DMA allocation or mapping into an object that doesn’t live long enough.

[0] The mapping operation may or may not be a no-op given IOMMUs, various VM translation schemes, etc.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#25

Post by quietbritishjim »

jandrewrogers wrote: post #11
quietbritishjim wrote: post #9

Earlier quoted context omitted.

> Making “move” and “destroy” distinct operations increases the flexibility and expressiveness. No, it does not. It's an artefact of the evolution of the language and highly undesirable. Rust has destructive moves (and copies built on top of moves, rather than the other way round) and it's far cleaner.

Sure, if you never need to deal with actual low-level high-performance systems code. Just because this use case doesn’t apply to anything you do doesn’t mean it applies to nobody. This is the kind of attitude that undermines languages like Rust (which I use in my systems). A fair criticism of Rust as a “systems language” is that it simply excludes all the really difficult parts of being a systems language. C++ deserv…

> Sure, if you never need to deal with actual low-level high-performance systems code.

What are you talking about? It's perfectly possible to write high performance code in C++ without violating the basic idioms if its type system. How would that even help? It sounds like you had some past project where you just didn't have the imagination to come up with a workable design. That's not my (or C++'s) fault.

The irony is, your comment reads as though you're defending C++ against my criticism of it. But actually your assertion that you have to violate move semantics to get performance out of C++, if that made the slightest but of sense, would be more of a criticism than anything I said.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#26

Post by motorest »

jandrewrogers wrote: post #5
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Not speaking to the specific design choices here, but in C++ moved-from objects are not destroyed and must be valid in their moved-from state (e.g. a sentinel value to indicate they’ve been moved) so that they can be destroyed in the indefinite future. This is useful even though “destroy on move” is the correct semantics for most cases. Making “move” and “destroy” distinct operations increases the flexibility and exp…

> (...) but in C++ moved-from objects are not destroyed and must be valid in their moved-from state (e.g. a sentinel value to indicate they’ve been moved) so that they can be destroyed in the indefinite future.

I don't think there is any requirement for a moved-from state other than the moved-from instance to remain in a valid state. There is zero relationship between moved-from state and the end of the object's life cycle. You can continue to use the instance of the moved-from object without any concern other than being in a valid state, but that applies to all instances of all conceivable object types.

Focus on the problem that move semantics solves: avoiding copies in general, specifically when resources are transfered to other instances. Does instance lifetimes change? No.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#27

Post by [deleted] »

jandrewrogers wrote: post #11
quietbritishjim wrote: post #9

Earlier quoted context omitted.

> Making “move” and “destroy” distinct operations increases the flexibility and expressiveness. No, it does not. It's an artefact of the evolution of the language and highly undesirable. Rust has destructive moves (and copies built on top of moves, rather than the other way round) and it's far cleaner.

Sure, if you never need to deal with actual low-level high-performance systems code. Just because this use case doesn’t apply to anything you do doesn’t mean it applies to nobody. This is the kind of attitude that undermines languages like Rust (which I use in my systems). A fair criticism of Rust as a “systems language” is that it simply excludes all the really difficult parts of being a systems language. C++ deserv…

[deleted]

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#28

Post by willdealtry »

rubenvanwyk wrote: post #7

Seems cool but I have questions about Arcticdb - is Polars, DuckDB etc really so limited for data science analysis that it's justified to write a new library specifically for time-series analysis on S3 files?

ArcticDB is more concerned with storage and persistence than with in-memory processing, so it's complementary to Polars/DuckDB etc, rather than an alternative.
Post reply on HN