Earlier quoted context omitted.
What's difficult to understand about pointers isn't the concept of a pointer itself, or even * and &, it's the fact that working with pointers requires you to simultaneously understand different abstraction levels. While it's not unique to pointers, and it's in fact the case for most nontrivial programming tasks, what's unique about C is that pointers are so pervasive you can't really do anything if you don't underst…
In C, pointers require you to think deeply about the ownership and lifetime of any "allocated object" at runtime. How long does it live, who is responsible for the deallocation, how many pointers does your program hold to that object (dangling issues). Ultimately, it can lead to a cleaner design if these issues are taken seriously up-front.
Unless you decide you use libgc, presumably?