I got a CS degree from Stanford in 2008, and I had the same worrys that you do. I didn't think I would be "smart enough" to keep up with the courses and the competition.
It turns out that Stanford's CS courses aren't that hard -- I was able to grasp the concepts quickly, and keep up with everything discussed in lectures and in books.
So what makes Stanford's CS program stand out? It's a lot of work. A lot of work. The intensity is what made the program so successful for me. Once I finished a class, I knew the material for life, because I spent so much time studying it. This is a huge contrast to the whole "cram for a test and forget it the next day" experience from my undergrad.
(Footnote: I was in the master's program, but I took quite a few undergrad courses to make up for my non-CS undergrad. My undergrad was in Information Systems at George Washington University. IS was a bad choice because the courses weren't very deep, neither in technology nor in business. But once I figured that out, it was easier to just finish up and graduate than to switch majors.)