Counter to OP's experience, I've actually found sweets to be more impactful than alcohol. That is to say eating dessert after dinner will impact my sleep more than a drink or two, and the periods where I lost weight have been more closely correlated with the periods where I was eating very few sweets.
Of course with both, the delta between low/moderate consumption and a baseline of zero consumption is low - it's the excessive consumption that causes trouble.
Like OP I found the daily ritual to be pretty easy after a couple weeks, but like OP I also missed the social aspect (and this is why I've sort of settled on the opinion that giving up drinking completely is not worth it, at least for me). If I felt like I was experiencing a step function improvement in life quality I'd keep it up... but for me it just hasn't been the silver bullet it's made out to be.
I'm not really one to label one food ingredient as the cause of all health problems, but if I had to choose one I would choose refined sugar long before alcohol.