4" is marginally bigger in terms of screen size, but it's quite a lot more awkward to use with one hand only.
I also don't like the 16:9 aspect ratio of iPhone 5, it's too narrow in portrait and too short in landscape. 3:2 was so much nicer. I love iPad's 4:3.
Finally, it feels like there's too much bezel on the iPhone 5 (compared to say Galaxy S3 and Note 2).