You linked to artofmemory.com. Did you ever read Moonwalking with Einstein? The author, Joshua Foer, was a skeptic of memorization parlor tricks. So he decided to try it for himself and actually became the USA Memory Champion, accomplishing memorization feats that he thought were impossible.
The book debunks many myths, most prominently that of photographic memory. The author also expresses his opinion about some famous characters known for their supposed memorization skills. For example, he explains how one guy who claimed to be reciting pi from memory was actually using his hands in a way that evinced the use of a mental arithmetic method, not recall of a memorized number.[1]
But the book is very clear on the fact that you _can_ memorize thousands of digits of pi. And he explores some famous characters who apparently did have perfect recall of trivia. In interviews with scientists, and drawing on his own experience, he draws parallels between their behavior and the memorization methods he and others learn and practice. Which isn't to say those famous characters were charlatans, but rather appear to be people whose brains naturally and uncontrollably applied similar "mental tricks" that normal people can learn and use, albeit with concerted effort and in a more limited fashion.
It also analyzes why these seemingly magical abilities aren't actually as productive and useful as you would intuitively think they are. Why forgetting is actually a very important faculty and remembering too much can be a serious deficit. So if you're skeptical because you're reasoning that anybody with such memorization abilities would be like gods among men, then the fault is in your assumptions about the function and purpose and power of memory in our intellectual and social lives.
Another interesting phenomenon is people who lack episodic memory. It's especially interesting because the use of memorization techniques like memory palaces actually leverages episodic memory. I didn't really make the connection with the particular concept of episodic memory--nor did Moonwalking with Einstein or similar books I've read, IIRC--until I later learned about the definition and distinction between episodic and semantic memory. Episodic vs. semantic memory is an important and well-defined dichotomy applied in the neurological research community.
In a memory palace you effectively create associations between episodic memory and trivia. (That's not the whole of it, obviously.) For most people instantaneous recall of episodic memory comes naturally; but episodic memory is subjective and emotional and highly malleable, not as objective and concrete as trivia. For some people (perhaps most people) with perfect recall of trivia the distinctions between episodic memory and semantic memory are blurred. In fact, some of them don't even have episodic memory at all. They can remember what they had for breakfast 12 years ago[2], but can't recall (re-experience) how they felt at the birth of their first child, even if they were overcome with emotion at the time.
These things are interesting because they expand the horizon of what we intuitively think is humanely possible. But at the same time they circumscribe and limit those possibilities. It's all very fascinating. All the modern analogies we use for memory, particularly with computers, are just woefully inadequate. Even neural networks don't come close to scratching the surface of what's going on underneath. The _depth_ and _breadth_ of the multifaceted phenomenon we call memory is much more extraordinary than we believe. And it's tangled with our other intellectual faculties.
[1] The odd deception notwithstanding, it's still an accomplishment. Con artists aren't lazy people. This guy did a Slashdot interview years ago, I think. He was supposedly gay and on the autism spectrum, but in retrospect he may simply have been a sociopath. I've read several books on that, too ;) He may have been legitimately gay but he seemed to play it for sympathy--to show how different and difficult his life was. Sociopaths don't experience shame or embarrassment, at least not like we understand it, so they can be much more sexually fluid when they want to be. Stripped of cultural values and emotional content, sex isn't qualitatively different than shaking hands followed by a little euphoria, even if you prefer shaking hands with one sex more than the other. But sociopaths lie and excel at manipulation, whereas lying and manipulation are hardly characteristic of autism.
[2] Presumably, of course, as it often can't be proven. The accuracy of much other trivia can and has be proven, so there's little reason to doubt them. Contrast that with photographic memory, where it's been thoroughly studied by several researchers and not a single person has ever passed a test or even come close to suggesting the legitimacy of their claim. Not that those people are charlatans, either. Just under an illusion.