Sure, if you've got crazy simple requirements it's generally a cinch. Once you do anything remotely non-trivial (want to read from a file? use some third party library, handle errors in a consistent manner) then you start hitting into issues with file permissions, PHP version differences, modules not compiled in by default, etc etc.Although PHP suffers from this, I'm not sure it is alone. I suffer a lot of pain setting up Ruby environments (I don't think I've ever managed it the same way twice) and getting everything to play nice. Even Python, a language I am about as familiar with as PHP, has it's annoyances on this front.
I suspect the difference is familiarity; I can deploy a working PHP environment in my sleep, because I've done it so many times.
The other day I had to set up Java/Tomcat - literally the worst experience of my life...
Also, PHP is not helpful in local development. I think there's a built-in runserver in the newest versions, but for all of history, to develop PHP you had to install (and set up) apache and all your required modules locally.
You're thinking like a seasoned pro :) For a newbie there is xampp - a familiar looking windows installer configures the Apache/PHP/MySQL (and others) stack, then it disappears into the backround and, largely, can be forgotten.
This is huge for a new developer.
Think about it this way; if you had a class called "Learn to Write Websites", for absolute beginners, I argue PHP is a good initial choice. Because you don't have to worry about the environment, how to server it, command line, package management - which, say, Ruby would require. You just chuck in a brief overview of how servers work in general, and then get them working on some PHP scripts, with instant results.
I love this!
I like Ruby, of course - it's powerful, has modern standards, and Rails is brilliant. But for a beginner, and even moderate, developer it is a bit of a pain to work with.
With python and Django, I use virtualenv+pip to get a consistent environment, manage.py runserver to test locally. I can pretty much start developing everywhere (my desktop, my VPS, my laptop, that same laptop after I reformat it and install some new distro to test) with 3 commands.
This sort of thing (http://pypi.python.org/pypi/virtualenv) is non-trivial for a beginner.
I'm not trying to defend PHP too much here; I like it, but it is a horrible language on many levels (as are other languages).
It also just works, especially for beginners.