I agree, with the caveat that the person with relative pitch may not know that the particular color is "green", but
does know that-- given two different colors-- both will have the adjective "dark" in their names.
In that analogy, suppose a "mise en scene" teacher gives the students an assignment to analyze the use of color in a picture. Using music dictation classes as a guide here, what would happen is that the person with "perfect color" immediately labels all the colors in the scene with the correct crayon name. This intimidates all the "relative color" people because they can't do the mappings without reference to the crayon box. Thus, the "relative color" people assume they are at a lower skill level in their analysis than the "perfect color" person. (This is exaggerated by the fact that the "perfect color" student has the word "perfect" in their name.)
Once the students finish their mappings the teacher asks the "perfect color" student, "How would you describe the use of colors here?" The "perfect color" student shrugs-- it is a collection of all the colors which they correctly named. As a kid they got praise from the teacher for their amazing ability to effortlessly map colors to names. No one ever asked them about the relationship among colors. (Or, if they did, the student nodded along and then correctly mapped colors to names as necessary to do an end-run around the question.)
On the other hand, "relative color" students grew up getting color names wrong. Every color class teacher would come up with a different ad hoc color theory and ad hoc color exercise to explain to the student how to correctly map colors to names. Most of them were wrong, but the student noticed weird little successes in identifying the relationship among colors. In their quest to stop failing all the time, they started to inarticulately ask themselves about the meaning of color, enumerating all the relationships among colors that could identify to help them narrow down the possible right answers to perhaps only a handful of names.
Back to the mise en scene class. One of these "relative color" student pipes in. "It looks like a bunch of pastels." The teacher praises the "relative color" student and moves on to other exercises.
After class the "perfect color" student asks the "relative color" student how they knew to say the word "pastels." The "relative color" student starts to talk about different famous scenes that have pastels, a childhood memory they have of a pastel drawing that won a prize at school, etc. But none of that connects at all with the "perfect color" student, because they've only ever had to map colors to names. They've never considered the relationship among colors.
So now the "perfect color" student dreads going to class for fear of the teacher discovering the profundity of their inability to say anything of substance about color relationships. And the "relative color" students dread the same class because they have no idea that their lifelong work to that point considering the relationships among colors constitutes a kind of knowledge.
Finally, there are no "perfect color" students who-- through sheer curiosity and discipline-- took it upon themselves to systematically investigate the relationships among colors. The people who did that would have placed out of this mise en scene class.