You can certainly write web applications in Java. They're harder to deploy than PHP but easier to deploy than Rails, in my experience, depending heavily on how experienced you are at administering systems and how much Java enterprise coffee you want to drink.
Java is a mature language. Libraries exist for almost any task you reasonably want to accomplish. There are skilled Java-speaking programmers in your area. Whatever you doing has almost certainly been done in Java already, and it has probably already been documented extensively. Java is an opinionated language: it is of the opinion that your workflow should resemble IBM's more than it does 37Signals', and if that is true, you and Java will get along nicely together.
There, mandatory praise out of the way. Java is horrifically maladapted to certain classes of problems and certain classes of users. One class of problem is expressively operating on strings to produce other strings. One class of users is anybody with a team size less than ten. Many web startups have a small team which is primarily interested in turning strings into other strings: if this is you, Java is going to fight you every step of the way.
99% of web application development for a certain type of business is turning strings into other strings. Many web development languages/frameworks popular with startups give you very expressive options for churning out certain types of strings quickly and with little cognitive load. For example, in Rails, there is very vanishingly little work in taking a bunch of strings from the blog_posts table and turning them into strings which encode a blog post into HTML. And if you want to add special strings that cause comments to be loaded in an AJAXy fashion, that is quick, too. Want to make sure that the strings the user is commenting with include certain strings which look like email addresses? Add one line to your program, and bam, if someone puts in a bad string they get a red string telling them how to make it a good string. It is all string rainbows and string unicorns. If you fat-finger your one line, it is very possible that large portions of your site will instantly break, but that is OK because if you program like everyone else using Rails programs, your magic guardian angel strings will get your attention before you deploy, and after you fix your typo everything is rainbows and unicorns again.
There are many Java frameworks. They can all operate on strings and produce other strings. However, the experience of actually doing this is maddening: pulling off the above you-give-me-bad-strings-I-give-you-red-strings can be a matter of touching six files or more, often written in painstaking configuration formats. The frameworks assume you are guilty of incompetence and force you to repeatedly prove you are otherwise, the better to prevent idiots like you from taking the site down.
This is a problem for startups because web startups are not about turning strings into other strings: that is just what they do. Web startups are about finding a problem people will pay money to solve. That is where they ideally spend most of their efforts. Sometimes, those explorations require code to be written, and when code gets written it should be quick to write, quick to test, quick to deploy, and quick to rip out when you discover your users don't respond the way you thought they would.
Java as it is practiced in the real world looks at that list of requirements and sees Very Scary Things.