True, I was thinking this over and I can see where surprise plays a part in curiosity but I don't think it's the main driver. I think your example shows this because when I click on the link out of curiosity it's because I don't have a good prediction for what will be on the other end of it. If I were a simple neural net then the outputs I would produce when seeing that link would be mostly similar across all the categories I could predict since I've never seen this url and there is little in it that allows me to predict what type of content it leads to. I am surprised to see the link since I couldn't predict it, so that's a point in favor of this approach, and I am also surprised by the content I see after clicking on it although only mildly because I didn't have a good prediction to start with. You're right that after I click on it I have very little curiosity to find out more about it but I think that is the key difference. The rest of that site remains an unexplored place, I've only seen one page of it which it seems would be a very small fraction of all of it. So why am I not curious about the rest of it and more crucially, what
would make me curious to explore it? I could imagine someone with a particular love of mathematics being able to exercise their curiosity on that page but is that because they would be surprised by what they found after clicking on each link or entering a new sequence? What would drive them to explore it? I think that speaks to the problem, if you optimize for surprise then the ideal reward is paying attention to an infinite number of TV channels (or a page with a bunch of links you've never seen before leading to different pages you've never seen) but I wouldn't call that curiosity.
I'm trying to imagine the simplest case, say a button you could press and every time you pressed it something entirely random would happen, always guaranteeing surprise. It would have a great deal of novelty at first but after a while it would cease to hold your attention even though your prediction of what would happen would never be accurate. I'd bet that after a while you might even never bother to push it again. The only way you would be convinced to push it consistently would be a) if you were assigned a reward for pushing it e.g. money in which case it is a slot machine or b) if by pushing it you could somehow reduce your uncertainty about what would happen which would as a by product reduce your surprise.
Thinking about it this way, surprise is certainly a key element at first. It grabs your attention initially but it doesn't hold it. What keeps you focused on exploring the thing that surprised you initially is the learning process which involves reducing prediction error i.e. reducing surprise. So there is a tension between the two.
The combination probably makes for a good exploration strategy. Initial surprise, look for a learnable pattern and follow it until another surprise, maybe backtrack and try other familiar patterns until those are exhausted and then investigate each sequence that led to a surprise by recursing through these steps.
This would also explain the example where my curiosity was prompted by the unknown link but I was not motivated to explore further. The website wasn't interesting to me because it was too unfamiliar and I wasn't able to find any familiar routes to explore through it due to my lack of interest in that area of mathematics but our hypothetical mathematician with a fondness for integers would see lots of familiar patterns they could explore attached to which are likely some enticingly unknown and surprising links.
Thanks for the prompt to think about this more!