1. Get a hobby, travel, volunteer, all that crap: No. That will not solve the inherent issue and as a result you'll still be lonely and unsatisfied.
2. Therapy: lol. Just no.
3. Religion, Pet, Marriage: Will make you more miserable. Now, not only you are lonely, but also have obligations (taking your dog pop).
Here is what you wrote:
> my neighbors from above, probably college students, are having a party of their lifetime, laughing, kicking and screaming
-> Fair enough.
> I was broke, renting a big flat with a bunch of roommates, but young, healthy, and full of enthusiasm (and booze).
-> That's nostalgia for a previous era. Nothing suggests you'll like that lifestyle now.
> I thought my life would only get better. And I made a good career, but in the process, steadily drifted off into isolation.
-> An important part that everyone here is missing. Let's tag this as "Exhibit 1".
> they're all full of lonely developers and other socially awkward people - short guys looking for girlfriends, unattractive/aging women looking for husbands, etc.
-> Everyone here is blaming (and judging you for judging others). Attractiveness is relative, blahblabla. Nope. Old tech people are both unattractive and boring, that's for both sexes (or all sexes). That's reality. You can sugarcoat it in your brain, but that doesn't create "boobs" or make a man stylish.
> All the "normal" people I knew in the past are now changing diapers and working to pay off their mortgages, which I guess is a kind of consolation.
-> Marriage is basically that. A process to make you forget, just like religion. It doesn't give you a meaning, it just makes you busy so you don't think about it.
> Doing things by myself is boring and depressing, and getting into a group of "normal" friends, of both genders, to hang out and laugh with, seems like an impossible goal.
-> Exhibit "2".
> In case someone has any advice beyond finding a hobby, going to therapy, approaching random strangers, it would be greatly appreciated.
-> You have no interests in any hobby (otherwise you'll be doing it) and please don't creep strangers.
> On the other hand, I feel that finding a non-coding job would be ultimately the best way to find new social groups, and get out of the apathy
-> No.
==
Okay. Take Exhibit 1: You had an idea, growing up, of where you wanted to be in the future. You thought that by building a good paying career, you'll have the lifestyle you "deserve". You succeeded in part I (career) and now are surprised that life is not as exciting and rewarding as it should be. You are in "deception" mode. It's like paying for a very expensive car to only find out that it doesn't work.
Take Exhibit 2: You are boring. You don't even like yourself (spending time alone). Of course, people are not going to like you.
So how do you fix this. First one, you should accept your situation. You are deceived but nobody promised you the moon if you did build a career. That was your own thinking. You need to get over it and realize that the time has gone and you should move on.
Second, you should become interesting yourself. Surprise: You might be interesting already without you recognizing it. Do you have an inherent interest in finance, politics, geography, history, music? You can strike a conversation about these topics for 2-3 hours and still talk about nothing.
Here is the last bit: The place, time and circumstance doesn't matter. You only need one friend with whom you share an interest in one of these topics. You share a beer/coffee once or twice a week. You discuss the latest rate hike, stock bubble or why Hitler lost the war. Now, suddenly, reading a book is interesting because you are building ideas for your next talk/discussion with this hypothetical friend. Bonus point if this friend challenges your ideas, do some research himself or is hard to convince in an intellectually stimulating way. Topics like finance are good because there is always something "new" happening in the market. So the discussion carries on.
There is nothing magical about life. Neither there is an end goal by itself. Your brain is just torturing you. You can "run away" by traveling to far away lands to forget about your deception, cold weather and problems. Or you can have have a beer a couple times a week. Second choice seems easier to me and you'll still have your finances in order. You can always travel later, but then you'll make a more informed choice and avoid another "deception".