We saw examples of simple quantification of people and activities, such as using counts of likes, stars, commits, and papers, and even more informed metrics like H-Index can lead to strange outcomes. AI will make the world even more quantifiable and, in many cases, falsely quantifiable. Ever since the first ape held two sticks in the left hand and three in the right and wondered which was more, ranking things by quantity is in our nature. The ape’s descendants have now discovered a ranking hammer, and everything will look like ordered lists. Ordered lists bring legibility, and what is not legible cannot be governed and subject to value extraction. The false quantification and rank ordering of things using AI will bring real-world weirdness in how people function, which has nothing to do with the functions they carry out. I call this the “Great AI Weirding”.
This reminds me of what Baudrillard terms "the precession of simulacra," in which successively more abstract representations of reality (from crude maps, to "hyperrealistic" GTA V-esque video game maps that are sometimes "more real" than reality itself) end up supplanting and taking place of the real. We no longer have people pursuing interests for their own sake (as per the "mathematician vs. mathlete" distinction made in the OP), but merely to construct a digital simulacrum of themselves, one which is able to inflate all the right metrics (there is a digression to Goodhart's Law [1] here) and win the same mechanistic games that we use as a proxy to measure value or worth in the world.
Ceci n'est pas une pipe. [...] All of these things have gone beyond what they point to.
That's it; we no longer have real pipes, but only abstract symbols and depictions of them. Having "precessed" past the era of when symbols were meant to point to, refer to, an underlying referent, they have become objects, referents in and of themselves - objects partaking of a purely abstract, symbolic reality. Instead of taking the pointer as a clue to investigating the nature of the referent, we accept the reality of the indirection itself; anything underneath our numerical abstraction is simply an "implementation detail."
In other words, they get huge information satisfaction from ads, far more than they do from the product itself. Where advertising is heading is quite simply into a world where the ad will become a substitute for the product, and all the satisfactions will be derived informationally from the ad, and the product will be merely a number in some file.
- Marshall McLuhan, 1966. https://www.youtube.com/watch?v=bNxo7fK-MJs
Consider this substitution: "all the satisfactions will be derived informationally from the [social media profile], and the [person] will be merely a number in some file." And yet of course, if you are "illegible," inscrutable, with little to no digital media presence nor statistics on your past "RL Career Game" history and performance, are you competent at all? Do you even _exist_? Does Harry, mathlete-turned-mathematician, even understand mathematics? Where is his Olympiad performance history? "[...] he became useless at competitions?" Oh.
I have recently been watching John Vervaeke, assistant prof at UofT in the fields of cognitive science and Buddhist psychology, and his lecture series "Awakening from the Meaning Crisis," where he describes the phenomenon of cognitive fluency:
When you increase the ease at which people can process information, regardless of what that information is, they come to believe it as more real, they have more confidence in it, etc.
- John Vervaeke, "Continuous Cosmos and Modern World Grammar," Awakening from the Meaning Crisis, 2019. https://www.youtube.com/watch?v=C1AaqD8t3pk
We have increased the ease at which people can process information, _about other people;_ and regardless of any correlation between the "quantified self" or the person's metrics, and the person-themselves, we come to believe that simulacrum of the person more real, develop more confidence in the constructed persona they project and their capabilities, etc. Conversely, a dearth of information regarding an individual makes them "illegible," somehow fictional, less real.
For all this, it takes a great leap of faith to object to playing these kinds of meaningless abstract games, at great personal risk and cost to one's self; yet I am not sure how to meaningfully participate in these systems without upholding and lending implicit assent to the fictions that they rely on. I am reminded of some meditations on Moloch regarding the matter.
One hope I have from Vervaeke's series is in his exploration of the notion of shamanism, and their role in society as developing new psychotechnologies and disrupting civilization's facilities for pattern recognition - altering their sense of what is important, altering their sense of selves, and altering the very way we think in the world. I look forward to a revival of the shamanistic tradition, applied to "cyberspace," (heh) to help us navigate the ways in which digital technology has altered our senses of meaning, what is actually important, and indeed of self and identity.
[1] https://en.wikipedia.org/wiki/Goodhart%27s_law