I've seen claims in both directions. To play it safe, I'll keep my 3G on 3.x. There was only 1 new feature in iOS 4 (that also worked on 3G models) that attracted me anyway: the unified mailbox view thing.
Actually I have a short list of desired improvements that might make it worth it, despite a speed drop:
* photo synching to my desktop (so not limited to the lame select-up-to-4-then-email-them thing, which is way too time-consuming if you have many photos and/or add new ones frequently)
* in Safari, when hit Add Bookmark, and goes into the Bookmarks view, it should start in the top-level collapsed state, that way I can quickly navigate to the right folder to select. Divide and conquer would speed it up in the case where there are many nested folder categories.
* making the Back button in Safari take you back to a cached, post-rendered view of the previous page -- rather than having to re-fetch/parse/render all over again, wasting my time, battery and the network (groan)
* cached application state restoration in general, consistently -- i realize the critical ingredient here is partly having enough memory and flash, so just beefing up the hardware with each new model refresh should get us to an ideal state eventually. Right now, lots of frustration and wasted effort when I go back in Mail and Safari and it's forgotten where I was last at and has to redo everything, sometimes reseting to some default state (such as scrolled to the extreme top rather than down in the middle where I was)