My experience may be of interest to you.
I first used LSD around 15-16. It was fun in that novel kind of way. Not life-changing.
Around a year later I moved to a new place where LSD was much more accessible and much cheaper. Myself and some new friends started using it frequently (i.e. every 2-3 weeks). I was also smoking pot regularly at the time. Some months later I had become complacent and we decided to go into the city centre on a busy Friday night while under the influence. I had a very bad time. Bad set and setting as they say. Strong paranoia set in - not unlike a lot of people would be familiar with through smoking pot, just much more intense.
That bad trip stuck with me. It really was life-changing. I developed strong anxiety, especially in social situations. It was as though, during that trip, my brain had learned some new associations and they persisted.
I have a large extended family that includes some with bipolar disorder and one schizophrenic. Knowing this, I was privately very concerned for myself. It took me roughly five years to learn to overcome that anxiety (no drugs or counseling, largely just self reflection). As with most things in life, I don't regret it as it is a part of what made me the person I am today. I know that I missed a lot of opportunities during those years, but I appreciate that I understand myself in a way that I otherwise might not have. I actually feel that I have come out psychologically stronger - able to recognise what is triggering certain feelings and having simple strategies for dealing with them (or even avoiding them in the first place).
Part of that whole process was learning that pot wasn't for me (fun as it was initially). Today I don't actually enjoy it at all.
Surprisingly, I do occasionally use psychedelics these days. I feel very safe doing so and I put this down to the hard work I put in following that initial bad trip 20+ years ago.
What I get out of psychedelics now (largely psilocybin and occasionally LSD) is very different from what I was looking for when I was young. I don't seek out an experience with strong hallucinations in general. I'm there for the psychological insight. I relish the opportunity it allows me for 'objective' self reflection. I know of no other way to be as honest with myself about my self. I get real insights that I later reflect on and that really help me get the most out of my meditation practice.
There's a question that you might have come across sometime. "When did you realise that you were the arsehole?" When I first saw that, I didn't give it much thought. Now I do. It's just part of the development of the self. I now welcome those moments where I catch myself being selfish, or reflexively and narcissistically defending the (always) inadequate conception I have of my self.
Psychedelics for me are a useful tool for honest introspection. Sometimes it can be exquisitely uncomfortable and you need to be prepared for that but I find it invaluable. Psilocybin seems to be better for this than LSD. It is the only drug I have ever experienced that leaves me feeling noticeably better the following day (I have similar thoughts when using LSD, but somehow it leaves my brain feeling a bit 'frazzled').
TLDR; When younger I chased strong 'fun' hallucinations using psychedelics and paid a price that taught me an important lesson. Today I use them for self reflection and the way they (ironically) help me to maintain and conceive of a more realistic self.