Several years ago, I decided to quit coffee as an experiment. This was provoked by hearing from various people that they'd quit and felt better, in particular that they no longer experienced sluggishness in the mornings. I was drinking the equivalent of maybe 3 or 4 double espressos per day, and nothing else containing caffeine, no tea, no soda.
So, I quit. Maybe a mild headache for a day, and feeling a pang of regret first thing in the morning. But that passed, and I stuck with it for several months without any issues and without difficulty. Maybe 6 months, I can't recall now. Perhaps the peaks and troughs in my energy levels flattened out a little, but it was so insignificant that I didn't feel any benefit.
Then one day, I thought I'd try a coffee again. Why not? So I ordered one at a café one morning on my way into work. After a few sips I felt what I can only describe as joy. I immediately wondered why the hell had I been denying myself this wonderful experience.
I'll never quit again.
As for anxiety, yes, drinking excess coffee can increase anxiety levels for me. I remedy that by mixing decaf (oh the horror) and caffeinated coffee if I want to keep drinking but I'm wary of the jitters. I brew fresh coffee in a mucha and I mix 1 scoop each of ground caf+decaf coffee. Taste is affected, but it's still pretty good.