Earlier quoted context omitted.
But Unicode code points aren't supposed to represent semantics, they're supposed to represent... well, "abstract characters" that are either glyphs or get combined into glyphs, which can have multiple (ambiguous) semantic meanings. That's why there aren't two different period characters to represent the end of a sentence vs. a decimal point, or two different em dashes where one represents a pause while the other come…
Thanks for shedding a little more light. Ignoring the semantics, in this case, between an apostrophe and a right single quote has resulted in many documents containing information that can not be parsed unambiguously because we have to pick a glyph and doing so with an ambiguously defined glyph loses contextual information. As a side-effect, since GPTs are based on the examples we give, they can't encode the proper p…
Well, and I'd suggest the unambiguous information was usually never there in the first place. It's less of an encoding problem, and more of an input "problem". People type either a single quote/apostrophe, or a double quote, and let smart quotes sort it out.
And sure, smart quotes will fail spectacularly with your spectacularly pathological example! Heck, it took me a few seconds to figure out what on earth was going on with the first 5 characters. :)
Your example would usually be typeset properly in a physical published book because it's done with professionals manually reviewing the typography.
Just throw it in the bucket of hyphens vs. minuses vs. dashes em and en, x's versus multiplication signs... our symbols are full of ambiguities, it's not just apostrophes.