No. I'm referring to the constitutional right to free speech.
The statement "anybody is free to do whatever they intend to in Australia, so as long as it doesn't impair other people doing so" is as absurd as it is false.
There are reams of legislation to prevent you from doing everything from arriving unlawfully by boat, to lying about your income, to using a wire carriage service to cause offence.
As to speech, here is a simplistic example: Let's say I walk into Federation Square and say to you "Go and burn down the Parliament, you useless white cunt!" Depending on who takes offence to my comments I can be:
a) Issued an on the spot fine for AUD$238.50 for using language likely to cause offence in a public place.
b) Prosecuted for racial vilification.
c) Prosecuted for sedition.
The fact that all but a) are highly unlikely does nothing to progress the case that "anybody is free to do whatever they intend to in Australia".