Vibrate on Silent is not silent at all

The iPhone volume interface has seriously confused me. There's volume, then there's alert sounds. The buttons only control one at a time but that depends on the last context you were in..