By far the biggest speed complaint I have about Firefox is not in everyday use, but whenever I restore a previously saved session - it basically stops reacting for a few minutes(!) before eventually I can use it again. I suppose it's due to the anti-virus interfering with some kind of memory image or whatever but whatever it is, it's so annoying.
Are you a tab hoarder by any chance? I get 2-3s of lag when restarting with a few dozen tabs open, but nowhere near "a few minutes".
I've probably got 60+ tabs open at any given point in time, spread out over multiple windows.