Earlier quoted context omitted.
I think what you were seeing was that many "nerds" don't see the term as particularly derogatory and even self-describe as such, though I suspect several of the congressmen did mean to be derogatory in their use of it.
It's derogatory because they used it in a way to imply that they didn't particularly want to be associated with the group labeled "nerds." I've always preferred the term "geek." Anyone else feel the same way?
Seriously, though, for a group of people who like to call each other out on standards of conduct, it's pretty appalling that they used a derogatory term to jokingly dismiss their utter lack of expertise in the area.