I use a mix as well. I think plaintext lacks three qualities that are very important to me:
1. Discrete structure/form. Yes you can enforce this yourself, but it's not inherent to the system. This means it takes some additional layer of work to group and move items around (e.g. of trying to keep in a prioritized order).
2. Free-form drawing/diagramming.
3. A built-in notion of time.
All of these are very important to me. Being able to group and shuffle the order of discrete items like todos is very nice for categorizing and prioritizing. Arranging future concerns on the basis of time (I have hard-time, soft-time, and any-time sections of my lists) is important for forecasting. And being able to freely draw is really important for things like architecture and logic diagrams, or UI mockups, or anything abstract that has structure (like a graph or list).
Where text really does shine, IMO, is logging/journaling and capturing knowledge bases or project details. Being able to just dump that stuff somewhere in a stream of consciousness manner and search it later is very useful.
My personal system is to use a pocket-size Moleskine (soft cover) and Microsoft Office tools, which is nice because I run Linux but have to deal with Word and Excel junk all the time anyway. Excel is great for organizing tasks in a GTD-ish manner and moving items around in a discretely, Word is good for checklists and knowledge bases, OneNote is a solid journal, and Outlook is a good calendar for week-ahead type planning. They also all support photos, so I can snap pictures when I want to persist things from my physical notebook such as diagrams or ideas.
I really don't think there's a silver bullet when it comes to tools or systems, so you should start from a fine-grained understanding of your needs and work from there. But plaintext as a format has pros and cons like everything else.