From my experience most of public networks won't let you do much this way. However, it seems it (as most captive portals) has access to DNS servers.
There was this tool people were using to bypass VPNs blocking and throttling in China called kcptun. It was letting you tunnel tcp traffic over udp, then SSL tunnel on top of it. With a server listening on port 53, it was working awesome to avoid QOS and managed to 1- bypass authentication and 2- get absolutely amazing speeds on some airport wifis for example. You probably could do the same with an openVPN on UDP 53.
However, it seems most public wifis are smarter and would blacklist your Mac address if either too much traffic is going through, or you say for too long. You can change your address, but it's not really usable. Still fun though!
Also, it seems most public wifis now do more DPI and they won't let other traffic than DNS go on UDP 53.
This in minds, another one I haven't been looking through much is DNS tunneling - would love to hear anyone's experience about it (I've heard it's very slow...)
Edit: seeing a few comments about the unethical aspect of this. In some cases, it might be. In some others, it is about avoiding a system that tracks you and try to gather and resell as much information as it can about you (it varies a lot according to which country you're in).