I get the sense that I'm old school compared to a lot of hacker news folks. As such, the way I've done it may not help but what the heck, can't hurt.
I listen to people, especially customers and potential customers, and pattern match. Been doing it for years. I try and get them to tell what problem they want solved but they rarely do that, they tell me what they want to buy. It's up to me to work backwards to what problem it is they want solved. Key point: listen to them, but not to what they want, find the problem they want solved.
Once you work out the problem, just shove it in a mental hash indexed by problem name and yielding a counter. Each time you see a problem, bump the counter.
When you are looking for a problem to solve, look through the hash at the ones that have high counters. Pick the one that you think you might be able to solve, looks interesting, looks like people will pay for it.
Believe it or not, that has worked for me for years, extremely well. It's perhaps hard to see that if you are in school or new in the industry. I remember what that felt like, it felt like I didn't know what to do. I remember thinking I couldn't come up with a good idea for a PhD thesis. I went to Sun and within a year or three had literally dozens of ideas, any one of which would have made (IMO) a fantastic thesis.
I'm not sure how to apply this to today's world. It worked for me because I went to work for a large company that had lots of potential customers. Marketing figured out that I liked to talk to customers and customers liked to talk to me (I was in the kernel group at Sun). I learned to keep a clean shirt hanging on the back of my door because I'd get random calls saying "Hey, lm, ATT is at PAL-1, can we swing by and get a brain dump on the clustering stuff?" which would lead to wide ranging discussions of what they wanted us to build. Access to those customers was huge, they were the source of the problems that needed to be fixed.
TL;DR summary - listen to customers, work backwards to the problem they need solved, build up a list of those, work on the one with the most hits.