Distinguishing those cases is the trick, I suppose. Especially for first-time offenders, there are people who
might be deterred from future offenses by being put on probation, once they've realized that they got caught and had better knock it off, who would instead turn into more hardened criminals if they're put in jail (and, with the current state of U.S. jails, likely end up joining a prison gang). But there are other people who have to physically be restrained from committing more crimes.
The current common sentence lengths don't seem that useful to me, though. Sentences like 6-36 months are just about the worst I can think of, keeping someone in jail just long enough to inculcate all the negative effects, while not actually keeping them off the streets that long. It creates this odd revolving-door mentality, where in some communities it's completely common to be in and out of prison, and everyone's got a bunch of criminals in their social circle. Now we could just lock all those same people up for 20+ years and not let them back out of prison to reoffend, but that seems like a problematic solution as well, given the current numbers (you'd have to be prepared to permanently lock up a large proportion of the population).
One start might be to start basing policy on empirical data. Can we find patterns in which kinds of policies have deterrent or anti-deterrent effects on which kinds of people? My impression is that laws and sentencing are not very strong based on empirical research currently. It does occasionally get brought up in debates, but the stuff written into the laws seems mostly pulled out of a hat.