Compared to my first gen SE, I'd like the faster processor, better camera and better battery life. But I'm not sure I'm willing to go up in size and weight, and get the rounded edges that make it even harder to grip one-handed.
For me, a bigger screen isn't a plus - after years of following the trend and being on my phone more and more, I managed to kick the habit and I'm very happy that my phone has now found its rightful place as a utility when I'm off my computer, rather than as a primary computer. I use it for maps, e-tickets, answering the occasional im. But it's not as convenient as my computer, it's not for leisurely browsing or media consumption, and I don't want it to be - it means that when I'm off my computer, I'm really off.
So this push toward bigger screens runs counter to what I want a phone to be for me. A bigger screen invites you to grab it with both hands. A bigger screen invites you to watch more stuff on it, watch movies, browse reddit and whatnot. I don't want any of that, I'd in fact prefer if it wasn't good at that, just like I don't want more sugar in my food.