The biggest factor, I'd say, is how frustrating a game is and how easy it is for someone who doesn't regularly play games to pick up. If a game requires a lot of hand-eye coordination, a lot of repetition, a lot of dying, it will not work. Period. I have no doubt you can do tons of studies to prove it. People like my sister are not interested in trying to find the skeleton key inside the blue ruby inside the giant book inside the treasure chest so that they can open some door to get a battery to operate an elevator so they can continue the game. The more nonsensical the gameplay, the less non gamers are interested in it. This pretty much eliminates 95-99% of videogames as frustrating garbage (and let's face it, for people without controller skills, they are nothing more than that).
It's amazing to me to see how many games fit into that remaining 1-5% of games and how fun they can be. I'm not interested in being frustrated anymore either, so other than sports games, those are mostly the only games I play or have played in the last few years. Some companies are realizing this and making great games for a variety of people. Others are making the same old garbage that even videogame veterans like me are frankly sick and tired of. Then they wonder why people hate their videogames. Like with most things, the answer is obvious.