In Go, pointers (mostly) don't go with slices in practice

Despite some syntax glosses in Go making it superficially sometimes look like a dynamic language, it is actually in the C heritage of what a variable is. A variable is not just a label pointing at an arbitrary value that can be moved to other arbitrary values like it is in most dynamic languages (like Python), it is a specific chunk of memory. (Or, more precisely, it is something that will be a specific chunk of memory if anything ever needs it to be, or the compiler decides it needs to be; recently things can in theory be register-only, but this is a detail the Go programmer operating at the Go level need not worry about.)

Specifically, what happens in

     s = append(s, "something")

is that append generates a new slice value (the tuple of backing array pointer, length, and capacity), which may or may not be pointing at a new backing array. These three values are then copied into the memory "s" was allocated with. So there is a new slice value allocated, but it is copied back to the original storage, and since there's no way to "get in between" those two things, the effect for a programmer at the Go level is that s is modified in-place.

In Go, allocation is very important and it is always explicit, except this explicitness is sometimes masked by the fact that the := operator is not a simple "allocate everything on the left using the types of the thing on the right" operator, but "allocate at least one thing on the left using the types on the right (it is a compile-time error if there isn't at least one new value) but use normal equality for everything already allocated", which is very convenient to use, but can make developing the proper mental model for how Go works trickier. Arguably, there's some sort of design mistake in Go here, though the correct solution doesn't immediately leap to mind. (It's easy to come up with more abstractly correct options but they have poor usability compared to the current operator.)

Similarly, it can be easy to miss that Go, like C, cares deeply about the size of structures, and every = statement has, at compile time, full awareness by the compiler of exactly how much memory is going to be involved in that equality statement. Interfaces may make it seem like maybe I can have an "io.Reader" value, and first I set it to some struct that implements it with a small amount of RAM, then maybe later I can set it to a struct that uses a large amount of RAM, but the interface value itself is actually a two-word structure with two pointers in it that is all you are ever changing, and, again, those two words are given a specific location in RAM (possibly virtually, if you never use it they could conceivably never been out of a register, but the Go compiler and runtime will transparently make it live in RAM if you ever need the address for any reason) and any setting of the value of the variable that has an interface value will set only those two values, with no other RAM changing as a result. You can use the same io.Reader variable through its interface implementation as a "handle" on a wide variety of differently-sized values under the hood, even in the same function (I do this all the time when progressively "decorating" an interface value within a function), but the in-memory size of the handle itself never changes no matter what value you ask it to handle.

This is not intended as criticism, praise, defense, attack, or anything else on Go itself; it is descriptive of what it is.

In Go, a []int ("slice of int") is just a C struct like this, passed by value:

  struct intSlice {
      int* addr;
      int len;
      int cap;
  };

The memory at addr is not owned by the slice. All the slice operations are simply notation for manipulating the struct. Go's garbage collection makes the whole thing work well.

This can be confusing if you're used to C++'s std::vector (which owns the memory) or Python's slices. Go's slices are a shallow pointer/length system exactly like is used in C all the time. For example:

  void sort(int* addr, int len);

becomes

  func sort(a []int)

A Go slice is just a formalization of C's pointer/length idiom, with terse notation for manipulation.

Too late to edit, but I made a significant mistake in the Go code: v[3] would always be a runtime error unless we explicitly modify *ref inside the function (ref = append(ref, 4), in which case v[3] == 4 would always be true).

This happens regardless of resizing, since append() at best modifies the array that *ref/v points to, but it does not modify *ref/v itself; and slices in Go have a pointer to an underlying storage AND a start and end index into that storage (multiple slices can point to different parts of the same storage).

Created here an example that shows how this interacts with resizing:

https://play.golang.org/p/UJ-t63bKyJJ

> They're also riding on the static typing trend that's happening right now, so type-safety is also part of the equation

I think the "static typing trend" is a product of Rust and Go showing people that static typing doesn't have to be cumbersome like it was in 90s-00s Java, C++, and C#. Indeed, I suspect that the quality of life improvements that Java, C#, and C++ made also improved the stock price of static typing (and building on that foundation, things like TypeScript are exposing JavaScript developers to the utility of types). Which is to say, static typing isn't an empty trend or fad (no idea if that's your intended meaning) but rather people were previously averse to static typing because the mainstream statically typed languages weren't ergonomic and people assumed that the bad ergonomics was caused by static typing--now we have many mainstream languages that show that this isn't the case.

> Honestly, this is a strange and peculiar situation, although Go programmers have acclimatized to it. To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea that surely should exist and work in Go. Well, it exists, and it "works" in the sense that it yields results and doesn't crash your program, but it doesn't "work" in the sense of doing what you'd actually want.

In C++, what happens is that the "iterators are invalidated" when you add something to an array. This is CONSTANTLY a source of bugs and frustration for new programmers. In C++, it may yield results or crash your program, and you are never sure quite which will happen. The best you can do (as a senior engineer) is design your software to avoid ever creating this situation in the first place and throw address sanitizer at things to try and catch them when they arise. The difference with Go is that in Go this will never result in a memory error.

Strictly speaking, the situation in Go is way better. I will take "incorrect behavior, but not a memory error" over "memory error" any day of the week.

We may forget what it's like for new programmers, but for those of us who hang out on Discord channels, Stack Overflow, and Reddit giving people help with programming, simple things like iterator invalidation are a major pain point.

"You have a memory error in your program", I say to someone. "Now that you know that you have a memory error, it is probably your highest priority to find and fix this error." And now you start walking someone through the steps of finding and fixing a memory error, which is nontrivial. You'll tell them about Address Sanitizer, GDB, and Valgrind, and you'll wish them luck.