I know little about the subject, but I'm sure there are differences between the limits of human height and the limits of biped primate height. If the giant of Castelnau were, as claimed, part of a "race of giants," it is not unreasonable to think this group explored the limits of biped height more profoundly, as a group, given that they could eliminate some (but not all) of the problems that height suggested.
On the latter point:
"There is no way a bronze age man with access to bronze age medical care was able to live long enough to grow nearly 3 feet taller than Wadlow."
You don't know that. Even in antiquity, with Iron Age medical care, men commonly lived to 100 years, with the absolute limit, as placed by the Etruscans, at 110 years, thus defining the Etruscan century at this number. Now, perhaps if the man in question was a giant there would be intrinsic difficulties in being gigantic, but natural selection within his tall genetic group could, as a group, overcome many of them. This is a very different situation that Robert Wadlow's, who did not come from exceptionally tall parents. If a group has selective pressure to become tall, it's an entirely different matter from an individual accidentally becoming tall.
Keep in mind there have been around 60 billion Homo Sapiens Sapiens, most of them outside historical record, and in prehistory, very isolated. This can commonly lead to vast phenotype differences. I would have to see stronger evidence to rule out that an adapted giant biped cannot reach 15 feet.