Obviously the thought of prison doesn't in fact stop people from breaking the rules. The prison population is in fact growing. Similarly many places have prisons that are very unlike the US prisons, and yet they have lower crime rates. So the evidence against the core of your argument is pretty strong.
At some point, when the punishment for minor offenses is extreme, the difference between minor offenses and major offenses gets very blurry. Extreme example: if stealing carries a punishment that is effectively the same as murder, the only logical thing to do when stealing is murder any potential witnesses. The US isn't quite this bad.
However, when the actual punishment for minor crime includes the concept of a felony record, it becomes much harder to differentiate between minor possession of drugs and say grand theft. They both carry felony record issues. That means the being convicted of either results in prison, exclusion from most of the work force for life, the inability to participate in a lot of "better yourself" programs (and really, people who messed up earlier in life should be encouraged to be in these!) and a host of other life long punishments. I understand this is a bit tangential to your point, but bear with me...
Now, when you look at the way the US justice system actually does things, you get this perception of arbitrariness or even randomness. The threat of prison is there, but there is no intuitive logic to how it works. Carrying some drugs will get you 20 guaranteed years in prison, while murdering someone will get you maybe 5, maybe 100. Raping someone will get you less time than "hacking". Racial discrepancies exist too in how prison is applied, as do wealth issues. There is parole/early release, and differences between judges and prosecutorial discretion further complicating the punishment question.
Whether or not there is good, deep, and sound defensible logic in how this works, the appearance is that there is no way of determining how you'll punished for what you're doing. As such, once the decision has been made to do something relatively harmless, the deterrent effect of prison no longer applies. The rational choice is "punishment, when it happens, could be anything", therefore the game is no about deciding actions based on punishment, it is deciding actions based on getting caught. Any action, no matter how severe, that lessens your chance of getting caught is better than a less severe action which doesn't modify or worsens the chance of conviction.
Basically my point is that when you want to use prisons to deter bad behavior, you better make it obvious that their application is something that can be reasoned about. As it stands, prison can't be included in any reasoning about consequences, except in an overly simplistic binary matter that has no effect at curbing any behavior once it is on the table.