My advice based on recent experience: give it time. We made domain changes and rankings tanked, only to recover. We made a major design change and rankings dived, only to recover. Give Google some time to assess.
Yeah, major design changes are painful. Google Webmaster Tools is currently warning me that we have over 100 404s on various URLs, because those pages simply got removed completely. Actually most of those 100 are the same few pages, but Google indexed it a metric shit-ton of times, and then added each of them to the index, and THEN it penalised us for "spamming." I came in, removed it completely, and set up a robots.…
And for a site thats been a round for a while only having 100 404's is low - most site owners are v bad at tidying up old defunct urls and using script kiddie platforms like joomla and magento dont help.