I hope you aren't downvoted for this, as this is basically my exact thought process for the last few years.
> make rich people richer
It is prop trading, so we are making ourselves richer, technically. We do a lot of charity work and donations with schools, etc.
> Doesn't it feel like a waste of your talents
Absolutely. However, I also like not having to beg my toddler to eat the food in front of him because payday is 10 days away, every credit card is maxed, and who knows how we will get to the grocery, which is what happened sometimes during my postdoc 3-7 years ago. Student loans were coming in at ~$1500/month when I'm making $2500/month. It was very difficult, and I was deeply depressed. When I finally landed that professor position, my workload got worse, and while there was a pay increase, I still wasn't going to be buying a house anytime soon. This was NOT an expensive area. We ended up living with my parents for a while (me, my wife, and 3 kids). The math just wasn't adding up, and there wasn't an end in sight. Adding in trouble getting grants, not wanting to restart the faculty job search, I started looking at industry. I applied at a few prop trading firms on whim, and mine ended up being a great fit.
I was pretty successful doing the actual research, but my particular niche (neuroscience+software methods and tools) isn't really funded by grants. I even got several rounds of good publicity for some of my PhD work, and did international interviews, etc. I was a very good scientist, but that doesn't cut it any more; new investigators are competing with established labs and members of the AAS who are submitting 30 grants per cycle. I was doing some very cool work with kids with epilepsy, which would greatly improve outcomes. I was a very valuable member of our collaboration; that research more or less came to a halt when I left, though I have been able to help out through consulting from time to time. So, what I did was in high demand and highly worthwhile, but there was a disconnect with getting the funding.
Am I being greedy? Maybe. There is that saying "you have to love yourself before you can love others." I think that might apply financially as well, particularly with regards to your family. It wasn't worth seeing my kids live in the stress of living paycheck to paycheck, going to crappy schools in crime-ridden areas that we can afford, with the added possibility of losing their dad because everyday he wonders if today is the day he drives off a bridge on the way home or something. If that means that my epilepsy research doesn't get done...then so be it. Vote for people who will increase research funding. So, when the offer came in and it was 3X what I was making as a prof, without considering bonuses, the weight of the world left my shoulders.