In Go, pointers (mostly) don't go with slices in practice • Hacker News Board

Re: In Go, pointers (mostly) don't go with slices in practice

#41

Post by _ph_ » Mon, Aug 30, 2021, 12:49 PM UTC

codeflo wrote: post #34

feffe wrote: post #32

Earlier quoted context omitted.

This is how I think about Go slices (may help other understand them). A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data. Typically slices are passed around by value but you can take their address and modify a "shared" slice. The built-in append() returns a new slice by v…

Does that mean that a) the pointer to the single element is only invalidated on “append” if the slice has no more capacity (this is how C++ vectors work), or b) does the fact that there’s an active reference into the slice always cause a reallocation (copy-on-write style)? If it’s the former, we’re literally in the C++ iterator invalidation nightmare, only without the debugging tools.

The array is only reallocated when it runs out of space. There is no magic about the "old" array. The GC just won't collect it, as long as a pointer to it exists.

I don't think there are many reasons, if any at all, to keep a pointer to an array element of a slice around in Go. Usually I only get the address of an array element only when passing it to some C code or doing some low level manipulation, but then I don't keep the pointer around. In Go code you usually just keep the slice object - which contains the necessary pointer anyway.

Re: In Go, pointers (mostly) don't go with slices in practice

#42

Post by _ph_ » Mon, Aug 30, 2021, 12:50 PM UTC

feffe wrote: post #32

friendzis wrote: post #10

Earlier quoted context omitted.

> This is safe, but confuses some people. Rust has created a weird perception that memory safety equals safety. Language is a tool and it should work with me: it is extremely important that my understanding of what the program should do aligns with what it actually does. The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container". Taking a pointer/reference…

This is how I think about Go slices (may help other understand them). A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data. Typically slices are passed around by value but you can take their address and modify a "shared" slice. The built-in append() returns a new slice by v…

Yes, this is entirely correct.

Re: In Go, pointers (mostly) don't go with slices in practice

#43

Post by dthul » Mon, Aug 30, 2021, 12:52 PM UTC

codeflo wrote: post #34

feffe wrote: post #32

Earlier quoted context omitted.

This is how I think about Go slices (may help other understand them). A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data. Typically slices are passed around by value but you can take their address and modify a "shared" slice. The built-in append() returns a new slice by v…

Does that mean that a) the pointer to the single element is only invalidated on “append” if the slice has no more capacity (this is how C++ vectors work), or b) does the fact that there’s an active reference into the slice always cause a reallocation (copy-on-write style)? If it’s the former, we’re literally in the C++ iterator invalidation nightmare, only without the debugging tools.

Without knowing much Go I believe it's neither a nor b. The pointer to the single element will always stay valid, no matter whether reallocation happens or not (and having a pointer doesn't influence whether reallocation happens or not). Re-allocation might be a confusing word here because afaik it's actually always a new allocation (the old one is not touched) and only if there are no more pointers to the old allocation will the next GC cycle deallocate it. So there is never iterator invalidation like in C++ but of course you still need to be careful because you might accidentally share or not share the same underlying data.

Re: In Go, pointers (mostly) don't go with slices in practice

#44

Post by _ph_ » Mon, Aug 30, 2021, 12:53 PM UTC

watt wrote: post #28

drran wrote: post #20

Earlier quoted context omitted.

No, ps still points to s: https://play.golang.org/p/iSjoqGTg20_O var s []int s = append(s, 10, 20, 30) pe := &s[0] ps := &s s = append(s, 50) s[0] = 100 pe2 := &s[0] fmt.Println("s: ", s, ", ps: ", ps, ", pe: ", pe, ", pe2: ", pe2) // s: [100 20 30 50] , ps: &[100 20 30 50] , pe: 0xc0000be000 , pe2: 0xc0000b8030

I don't get why address of slice returned from "append" does not change. Maybe in a trivial program like this the backing array can always be extended in-place, because there in memory fragmentation. Is that still true in an app that has considerable memory pressure and has GC running now and then?

Append returns a value, the new slice struct generated by append. Append always generates a new struct and returns it by value, because even if the array pointed to doesn't change, the length property of the slice changes. This value is then assigned to the local variable s, which didn't change its memory location.

Re: In Go, pointers (mostly) don't go with slices in practice

#45

Post by kubb » Mon, Aug 30, 2021, 1:06 PM UTC

Considering the confusion of the author, it seems like not all junior programmers can understand Go, which makes me wonder: is it simple enough?

One pitfall is when getting a slice by value in a function. You cannot be sure that someone is not going to pass you a slice into a buffer that they themselves use, so you have to be careful when appending - someone might be using that buffer and you’ll be writing over it.

Re: In Go, pointers (mostly) don't go with slices in practice

#46

Post by friendzis » Mon, Aug 30, 2021, 1:07 PM UTC

codeflo wrote: post #34

feffe wrote: post #32

Earlier quoted context omitted.

This is how I think about Go slices (may help other understand them). A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data. Typically slices are passed around by value but you can take their address and modify a "shared" slice. The built-in append() returns a new slice by v…

Does that mean that a) the pointer to the single element is only invalidated on “append” if the slice has no more capacity (this is how C++ vectors work), or b) does the fact that there’s an active reference into the slice always cause a reallocation (copy-on-write style)? If it’s the former, we’re literally in the C++ iterator invalidation nightmare, only without the debugging tools.

AFAIK, neither.

Since slice API is pass-by-value, in theory ANY method will invalidate the pointer. In practice only resizing methods actually NEED to reallocate the underlying array, but magic can happen. However, refcounting will make sure that a previously underlying array having pointers to it will remain allocated. This means that 1. pointers to single elements will always dereference 2. slice structure modification can leave pointers pointing to stale data

Re: In Go, pointers (mostly) don't go with slices in practice

#47

Post by leaveyou » Mon, Aug 30, 2021, 1:20 PM UTC

simiones wrote: post #23

pjmlp wrote: post #21

Earlier quoted context omitted.

Not really, if reallocation takes place. So if you got a pointer to a vector element, it now points to garbage.

I guess the difference is this: C++: std::vector v {1, 2, 3}; void foo(std::vector *ref) { ref.push_back(4); } foo(&v); //v[3] == 4 is true here Go: v := []int{1, 2, 3} func foo(ref *[]int) { append(ref, 4) } foo(&v) //v[3] == 4 may or may not be true here. Pointers to elements in the vector do indeed have the same problems both in Go and C++ (except for memory safety).

append(ref, 4) // error: first argument to append must be slice

append(*ref, 4) // error: append(*ref, 4) evaluated but not used

this is the correct version and it also removes the incertitude:

*ref = append(*ref, 4)

https://play.golang.org/p/-xDqaxvqWhm

Re: In Go, pointers (mostly) don't go with slices in practice

#48

Post by sly010 » Mon, Aug 30, 2021, 1:20 PM UTC

Go-s behavior is the ONLY sensible one in _any_ language that supports pointers. This is a faster and safe(er) way. You simply cannot modify (move or reallocate) a data-structure that has pointers pointing to it without invalidating all pointers. Not in C++, not in any language with pointers (that I know if). This is not "strange and peculiar". What's "strange and peculiar" is that the author thinks that doing this in C++ is a "perfectly decent idea". In fact it's a huge no no and more often than not will crash.

Edit: we would both learn something if you offered a counter example instead of downvoted.

Re: In Go, pointers (mostly) don't go with slices in practice

#49

Post by sly010 » Mon, Aug 30, 2021, 1:29 PM UTC

friendzis wrote: post #10

Animats wrote: post #3

Well, of course slices work that way. Think about what happens if you have a reference to a slice in an array and you shrank the array to 0. You've just created a dangling pointer. In Go, you get a stable version of the old data, and the garbage collector tracks that you still have a reference to it. This is safe, but confuses some people. In Rust, the borrow checker won't let you modify the array while you have a re…

> This is safe, but confuses some people. Rust has created a weird perception that memory safety equals safety. Language is a tool and it should work with me: it is extremely important that my understanding of what the program should do aligns with what it actually does. The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container". Taking a pointer/reference…

> it does wrong thing without warning

It's not without warning, it's a well documented behavior.

Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.

Mutating them in any way is actually a special case that you do only for performance reason (i.e. you can pre-allocate and fill if you know the size ahead of time) but - as always - you try to keep those abstracted away and to the minimum.

Re: In Go, pointers (mostly) don't go with slices in practice

#50

Post by dilap » Mon, Aug 30, 2021, 1:40 PM UTC

> To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea that surely should exist and work in Go.

Ah, I would beg to differ!

You should never be taking pointers to a dynamically resizable array, in any language. (Well, caveat, its fine if you do it only for a time period where you know the array won't be growing.) The whole point of a dynamically resizable array is that its addresses can change!

If you did this in C++, you'd get undefined behavior. In Go you get "safe" but probably-not-what-you-wanted behavior. In Rust it simply wouldn't be possible (w/o unsafe), and you'd have to use indices (which is the correct thing to do, in any language).