I traveled for a couple of years, then went back to get a PhD in mathematics.
Are you a professor or a research scientist now? I have pondered this route, but it seems that in general, the research community / academia tends to "shun" those that don't do straight undergrad -> research.
I dropped out of my undergrad CS program to do my first startup. That was 15 years ago. When I decided to go back for a PhD two years ago, I still had to complete my undergrad. I'm finishing up this spring and will be starting at a top 10 graduate program in the fall. My grades from 15 years ago were not particularly good and my background is rather unusual. I wasn't sure how I would be viewed as an applicant. This was not a easy year for admissions and I feel I did extremely well.
I know two other people went back to study math after working for a while. One is now a grad student at Brown, the other is a postdoc at MIT.