That is a reasonable response. I would counter by saying that I see that as a justification for why they [government, ISP, whoever] should be
allowed make demands about the use of the physical commons, but isn't a reason why they should exercise this conceded right.
For example: I, a hypothetical bar owner, have a right to ban silly hats in my bar. Why? Because I own it. However that's not a reason that I should ban silly hats. Just saying "I'm the owner, so I can." doesn't actually explain why I should.
I do also disagree that the government specifically is entitled to restrict privacy on the internet because much of the infrastructure is owned or otherwise controlled by them. The government owns nearly all roads, yet while using that infrastructure I still enjoy certain privacies. For example, if a cop pulls me over and asks me where I am going or where I have been, I have no obligation to answer him.