Some drawbacks:
running on hot pavement - the thin soles enable you to feel the temperature of whatever surface you're on. It never gets unbearable, but you'll definitely know when you're standing on a hot surface.
stepping on a single small rock - wearing normal shoes, the soles don't bend much if you step on a rock, but the vibrams flex around the rock so you'll definitely feel the pain.
stubbing your toes - the vibrams make it feel like you're barefoot, but the shoes make your foot slightly larger so sometimes you might trip on yourself walking normally because your body assumes you're barefoot.
not good on small clipless pedals on bikes - I bike a lot and sometimes I'll jump on my bike without my shoes with cleats. Normally it's not a problem with normal shoes since the rigidity of the soles protect me from the small surface area of the clipless pedals, but the vibrams make me feel like pedaling on tightrope.
getting sand in shoes - I have the KSOs, but sometimes a small sand particle will still make it in. Once in, the sand particle will make it down to the sole area and you'll definitely feel it since you don't wear socks to wear vibrams.
You'll get over most of these drawbacks after a few months though. The shoes are definitely worth it over the small drawbacks. If you're not sure, you should try them out in a running store first, you don't have to order them over the net.