It's worth understanding that the light which you see is the result of
many photons -- though it's usually estimated that maybe, if our rod/cone cells were a little more sensitive, they might have been able to perceive individuals. So
actual energy/mass transfer comes from a
lot of particles each offering their own individual kicks. So the overall intensity is not just (energy per kick) but really (number of kicks) * (energy per kick). The number of kicks counts.
Phonons, for sound, actually are quantized in the exact same way as for light -- just replacing the speed of light c with the speed of sound in the medium v. (Both arguments are based on analogies to a harmonic oscillator problem which every quantum mechanics course works out; basically any vibration in the bottom of a potential well X can be dealt with by approximating that potential well with a parabola, which leads to particles, we could call them X-ons, of one form or another. Quantum mechanics makes all particles act like waves but also makes all waves quantize into particles.)
So, long-story-short, the physics is the same, and long wavelength phonons also carry less energy per kick than shorter ones, just like photons. That's not the difference, and the E = h f equation is still the same. But for phonons it's more about numbers.
The biggest sound particles you can hear correspond to 20,000 Hz frequencies. The smallest light particles you can see correspond to 700 nm wavelengths or 10^15 Hz frequencies. So photons are more than 50 billion times more energetic than phonons, as a rule.
This changes a lot of features about them. For example, every degree of freedom in a system has some average energy due to just random energy-sharing, which is known colloquially as the "temperature" of the system. If you work out how many 20kHz phonons there are simply due to a room being at room temperature, it works out to about 300 million. Any detector which registers 20kHz phonons, including the hairs in your ear, have 300 million of those phonons in them just from random thermal excitation. On the other hand, the red light receptors in your eye have on average (300 million) / (50 billion) = 0.006 photons in them on average, meaning that if you look at a thousand of those photon-absorbers you'll see on average only six of them which are excited randomly just by thermal excitations. So we can, at room temperature, detect individual photons, where we can't detect even ten thousand phonons at once without it getting hopelessly buried in thermal noise. (But our eyes simply do not fire the neurons when a light-sensitive cell only sees one photon.)
