This may be true that the private system as a whole saves the government money. However, I'm not sure that the last dollar of subsidies spent on the private system saves the government any money at all - i.e. would someone switch to a public school if that dollar was taken away, or would the private schools just move the cost to the consumer who would bear it, or perhaps they might cut back on the indoor shooting range or second Olympic pool (in the case that the dollar is going to an elite private school)?
The argument about that last dollar obviously extends backward quite a way before becoming hugely disruptive or that big a deal. Obviously cutting all subsidies to the private system overnight would be an enormous disruption, but it's far from clear to me that the policy of subsidizing private schools at the current level saves the government any money at all.
As for keeping the government system honest - I think this works both ways. I think having people exercising "feet" rather than "voice" harms the public system by taking out people from participation. The argument is similar about selective public schools (I went to one). I see a lot of flaws in the public system too - tons! - and any strong movement back towards public education in Australia would have to be accompanied by some major changes in how they are run (IMO way too much for the convenience of mediocre teachers and even-more-mediocre bureaucrats).