tl;dr
It turns out having your framework written in c and loaded as a php extension makes it fast as truck.
Other than that, smaller frameworks tend to have faster load times and smaller memory footprints.
Stay away from Zend frameworks.
Watch PHP Developers Fight Over Framework Speed: Zend Is A Hog
reddit.com