I have read much of it. It's a project I started in my late teens, and have revisited many times over the subsequent twenty years. I have done many of the (simpler) exercises, and I have even sent in a letter detailing a small error in the text, yielding me a minor check (not to be confused with the major check) from the Bank of San Serriffe. I also got a response from Don Knuth, who corrected my letter inline in pencil. Both an awe-inspiring and humbling experience.
Should you read the series? Well, that's largely up to you. There are other books that cover subsets of the material, and many books that cover algorithms not in Knuth's text. However, there are such amazing gems of knowledge in this work that it would be a shame to miss out. I'm sure that many people keep this set on their shelves for some sort of bragging rights, but I often leaf through sections for both inspiration and for entertainment.
You don't _need_ to read TAoCP, but it is a refreshing read. To get the most out of the text, you will need to have a pretty good understanding of mathematics, and you will need to be willing to work out the things he describes on your own. It's not a passive reading experience, like reading most fluffy books found in the "computers" section of the average bookstore or library. It is an active experience, like reading a mathematics textbook. There will be parts that challenge your conceptions, and there will be parts that will take several attempts to fully understand. Then come the exercises, some of which are still open problems in Computer Science.
It is a worthy endeavor. I recommend it.