quite loaded words there. "knowledge", "relationships".
the llm must be able to output the correct code when the input says something like 'place object x to the left of object y' vs when it says 'to the right'. there is an absurd combinatorial space of the possible inputs vs outputs it must generate -> the whole set of these, interpreted from a human point of view, you could call knowledge. the [x] in input x output you could call the relationships. And the fact that the LLM doesn't have to brute force represent all of them (impossible in the limited embedding vector x internal representation state) you could interpret as 'understanding'. But again, these are loaded words that are somewhat meaningless when looking at what an LLM does in a literal way.
They have plenty graphics code to train from, so this structure will be directly or indirectly available in the training data in a very plentiful way.
In literal llm transformer terms, the embeddings must have some of their components statistically represent spatial structure in some way that later in the internal layers of the LLM give some statistics of how likely it is to occur for certain code tokens to appear relative to the input of spatial wording in the token stream.
it is likely that their internal representations encode something more general than specific input x output stream combinations (because we already know this is the case for regular words and concepts). if not in the embeddings, then a couple layers into the network for sure.