You misunderstand what a language model is. What a human can or cannot do is frankly irrelevant.
A GPT (Generative pretrained Transformer) is not a simulator nor an imitator. It is a predictor.
Predictors try to reverse engineer the computation that could have led to a certain output so they don't just grok what is explicitly stated in a dataset but also what is implied by its structure.
As an example, In a language model trained on protein sequences alone and nothing else, you will find biological structure and function emerge in the inner layers.
https://www.pnas.org/doi/full/10.1073/pnas.2016239118
In fact, with a capable enough architecture and sufficient data (quantity, qualit, variance), there is nothing a predictor couldn't divine with enough training.
You assert
>Actually generating a recipe is obviously much harder, and should be effectively impossible unless you include a humanoid robot that can cook and taste.
In the pursuit of recipe prediction, a predictor will learn to internally model taste because taste is implicit in the data.
You don't need the hypothetical perfect predictor to demonstrate your assertion as false because a capable predictor already does.
>If you're wondering whether a task is theoretically possible for an AI, I think a good rule of thumb is to ask "could a human domain expert do this given a few days and as much research material as they need".
You are wrong. Train a language model on descriptions of the functions of proteins and an equivalent protein sequence and you get a language model that can predict novel functioning sequences from function descriptions alone.
https://www.nature.com/articles/s41587-022-01618-2
Not only is this not something a human expert can achieve in a couple days, It's not something a human expert can achieve at all. It is a Super-Human ability.