The first problem is that there is a highly predictable and (at the moment) constant supply of bitcoins, which means that the value of the coin will adjust like a commodity as more or less people start to use it. It's like Gold or Oil in the sense that when people want it, it's worth a lot, and when people don't want it, it's not as expensive. That chains the value to the demand, meaning that any time you use bitcoin as a store-of-value, you are putting yourself at the complete mercy of the market.
Many advocates believe that as the number of people using bitcoin grows the price will stabilize, but I think that using bitcoin as a store-of-value is going to end up like using the gold standard, especially because some early adopters have tens of thousands to even a million (Satoshi is believed to have a million) bitcoins, and any of them could decide to 'cash out', which would dramatically change the supply and potentially affect the whole market.
Another problem is that the currency is highly traceable, and while you can do things to obfuscate your spending habits there are lots of techniques involving statistical analysis that shed doubt on the effectiveness of obfuscation. You can take this back to the "I have nothing to hide" argument, but there are plenty of powerful parties that will be more interested in a currency that can't be traced (imagine big business or big finance), and an untraceable currency would certainly be more attractive.
But to me the biggest current problem with bitcoin is the uncertainty. There are many conflicting schools of thought in economics each with their own reasons why bitcoin is good or bad. But beyond the basics, macroeconomics often involves a lot of voodoo, and for any new paradigm there will be major schools of thought that will believe the new paradigm is bad or unstable in some way. There is no real way to fight this except to accept that we ultimately have no idea how new economic paradigms will affect our world, and that some of the naysayers may be correct and thus caution should be used.