> GPT-4 is not a pure LLM
I’ve looked a bit into the GPT architecture and haven’t seen anything suggesting it’s doing special-case experts for maths. It has MoE over 16 language models, and an image modality bolted on. If you have any evidence that there is a separate trained logic/math model I’d love to see that, as it would be interesting. (I don’t recall reading anything like that in the GPT papers for example, and this seems to claim there is no “calculator” hooked up in GPT-4 https://ai.stackexchange.com/a/40090).
> To put it another way: a (pure) language model has no way of encoding numbers
I think you just motte-and-bailey’d. Your original claim was that a LLM was incapable of doing $X > $Y or displaying numeracy, which I refuted by showing an example of an LLM doing greater than comparisons, and subtracting a quantity in different units ($50k -> 50,000).
Now you are substituting a much narrower claim, that an LLM is stucturally incapable of symbolic manipulation and “really modeling numbers”. This might be so! But it’s not required for basic numeracy, “tricks” as you put it, or whatever else GPT has learned, can objectively get us to median human performance.
Even going way back to GPT-2 there are mechanistic interpretability papers investigating how greater-than is implemented, eg https://arxiv.org/abs/2305.00586.
And there is work that suggests that LLMs do some sort of phase transition to gain numeracy skills: https://arxiv.org/pdf/2206.07682.pdf.
Your objection about working memory is also odd. Chain of thought reasoning strategies use the context as the working memory and have been demonstrated to improve performance on numeracy tasks.
But again, if you are retreating to a very narrow claim that the model can’t do precise calculations in a single inference step, then sure, that’s technically plausible, but that’s a way higher bar than displaying basic numeracy, and doesn’t justify the incredulity in your GP comment.