"... we are all subject in the United States to the Patriot Act and it is possible that all that information could be made available to the authorities."
The right to privacy from the government is a logical consequence of the fact that the government is the servant of the people, put in place to protect and promote the people's interests. The Declaration of Independence describes our reason for instituting a government at all:
"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. — That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, ..."
We need no justification for privacy from the government and its proxies. The most upstanding citizen still has the right to keep his affairs private. Even people with nothing to hide have the right to hide it.
We explicitly give the government the power of search, but only when we (through a judge) agree that it's necessary. Without that agreement, they're not allowed:
Amendment 4 - Search and Seizure. Ratified 12/15/1791.
"The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."
We explicitly restrict all other powers of government through the 10th Amendment:
Amendment 10 - Powers of the States and People. Ratified 12/15/1791.
"The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people."
We do that because we recognize the tendency of unchecked governments to gather more power to themselves. When a government becomes more powerful than its people, the people are slaves who have lost "the Blessings of Liberty" as listed in the Constitution's Preamble.
Government is a dangerous tool. It's the people who should be watching the government, not the other way around.