Most experience developers will come up with the architecture/design of what they are working on through a combination of intuition and experience, write tests after and then use TDD to fill the gaps once they have nailed down the architecture.
I do TDD some of the time and my workflow often look like this:
1. Start implementing an object/class, defining APIs of its key methods while having a higher level picture in my head of how it will fit with other components
2. Implement a good chunk if not all of that object (not more than a few 100 lines typically)
3. Write a test to verify that what I did worked.
4. Start commenting out parts of the code I wrote to make sure the applicable test cases fail. Uncomment after verification is done.
5. Use TDD to cover and implement edge case handling and anything leftover
6. Repeat 1-5 for each class/component/module until done
I only get into TDD once the architecture and high level design are finalized, and I have some code written for them and I have some reasonable level of confidence that it is not gonna drastically change. I also generally go "bottom-up" starting from the leafs of the object graph up and use TDD maybe ~50% of the time (just a rough guess).
Trying to force yourself to let "TDD drive the design" will likely result in frustration as you have experienced and slowdown.