Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sparrow, a modern C++ implementation of the Apache Arrow columnar format

johan-mabille.medium.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#11

Post by jandrewrogers »

quietbritishjim wrote: post #9
jandrewrogers wrote: post #5

Earlier quoted context omitted.

Not speaking to the specific design choices here, but in C++ moved-from objects are not destroyed and must be valid in their moved-from state (e.g. a sentinel value to indicate they’ve been moved) so that they can be destroyed in the indefinite future. This is useful even though “destroy on move” is the correct semantics for most cases. Making “move” and “destroy” distinct operations increases the flexibility and exp…

> Making “move” and “destroy” distinct operations increases the flexibility and expressiveness. No, it does not. It's an artefact of the evolution of the language and highly undesirable. Rust has destructive moves (and copies built on top of moves, rather than the other way round) and it's far cleaner.

Sure, if you never need to deal with actual low-level high-performance systems code. Just because this use case doesn’t apply to anything you do doesn’t mean it applies to nobody. This is the kind of attitude that undermines languages like Rust (which I use in my systems). A fair criticism of Rust as a “systems language” is that it simply excludes all the really difficult parts of being a systems language.

C++ deserves a lot of criticism. Many aspects of the language are quite fucked. But willfully ignoring that it solves real problems that other nominal systems languages are unwilling to address doesn’t mean those problems don’t exist.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#12

Post by arka2147483647 »

amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Why does sp::primitive_array even exist?

Should it not be std::array?

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#13

Post by arka2147483647 »

jandrewrogers wrote: post #11
quietbritishjim wrote: post #9

Earlier quoted context omitted.

> Making “move” and “destroy” distinct operations increases the flexibility and expressiveness. No, it does not. It's an artefact of the evolution of the language and highly undesirable. Rust has destructive moves (and copies built on top of moves, rather than the other way round) and it's far cleaner.

Sure, if you never need to deal with actual low-level high-performance systems code. Just because this use case doesn’t apply to anything you do doesn’t mean it applies to nobody. This is the kind of attitude that undermines languages like Rust (which I use in my systems). A fair criticism of Rust as a “systems language” is that it simply excludes all the really difficult parts of being a systems language. C++ deserv…

Respectfully, the value of a c++ wrapper/implementation comes from the fact that it behaves like one would expect a C++ classes to behave. That is, RAII, and so on.

If the underlying resource can not behave like a class, it would be better to expose a free function style api, eg:

    Handle h = ResourceGet();
    ResourceDoSomething(h);
    h.release();

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#15

Post by BoingBoomTschak »

senkora wrote: post #6
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

This bothered me enough to check the source code, because I simply had to know: template std::pair get_arrow_structures(A& a) { arrow_proxy& proxy = detail::array_access::get_arrow_proxy(a); return std::make_pair(&(proxy.array()), &(proxy.schema())); } https://github.com/man-group/sparrow/blob/c01a768f590ebf3b22... So the answer is that the `std::move` does nothing and should be omitted, because this function only ha…

Modern C++ in a nutshell

t. C++17 dev and pain connoisseur

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#16

Post by binary132 »

I am glad to see others in the comments share my concerns about the code quality considering that the offered samples are not really idiomatic or modern C++. In particular, there is really no reason to use pointers here in either the accessor or the mutator; both of those probably should have been references, although the case can be made for the mutator at least since it makes it explicit in the written code that we may mutate the arguments.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#17

Post by JTyQZSnP3cQGa8B »

amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

It's a new technique called "reverse GC" where you care about the first living object instead of the last one /s

Also:

> nullable can hold references

We have: https://en.cppreference.com/w/cpp/utility/functional/referen...

> Assigning nullval [...] does not trigger the destruction of the underlying object

That's weird.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#18

Post by SylvainCorlay »

arka2147483647 wrote: post #12
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Why does sp::primitive_array even exist? Should it not be std::array ?

This is really not the same thing at all.

sp::primitive_array holds memory following the Arrow specification, which can be operated upon in place from e.g. ArrowCpp, PyArrow, etc.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#19

Post by SylvainCorlay »

juunpp wrote: post #3
amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

I think that comment is a copy-paste mistake. If you look at the next code snippet, the comment actually makes sense there. That being said, I've also given up on C++ and learn it mostly to keep up with the job, if that's where you are coming from. I don't find Rust to be a satisfying replacement, though. No language scratches the itch for me right now.

Excellent catch. The std::move in the snippet is a copy-paste mistake. It was carried over from the previous code snippet.

Re: Sparrow, a modern C++ implementation of the Apache Arrow columnar format

#20

Post by SylvainCorlay »

amluto wrote: post #2

This is supposed to be idiomatic?!? namespace sp = sparrow; sp::primitive_array ar = { 1, 3, 5, 7, 9 }; // Caution: get_arrow_structures returns pointers, not values auto [arrow_array, arrow_schema] = sp::get_arrow_structures(std::move(ar)); // Use arrow_array and arrow_schema as you need (serialization, // passing it to a third party library) // ... // do NOT release the C structures in the end, the "ar" variable wi…

Excellent catch.

This std::move should not have been in this code snippet. It is a copy-paste mistake, carried over from the previous code snippet of the post, and should have been omitted.

(The `std::move` does nothing in this snippet, since `sp::get_arrow_structure` takes and lvalue reference).

In the previous example with `sp::extract_arrow_structures`, which takes an rvalue reference, std::move is required and the sparrow primitive array cannot be operated upon after.

Post reply on HN