If you don't work hard, you probably won't be successful. But if hard-work were sufficient for success then the richest person in the world, or the President, or however we measure success would also the hardest working, which I don't think most people who agree with.
Similarly, although I think most sports-people would be acknowledged to be very hard-working, there is a matter of luck in their success - you need to find the right sport for you, you need to avoid injury, you need to perform at your best on the day of the big competition / when the talent-scout is visiting.
Perhaps it's just a problem of perspective - if you are relatively successful to begin with you can see that there are a lot of people less successful than you who worked harder by some metric and so would attribute success more to luck and connections than hard-work; if you are relatively unsuccessful, perhaps the correlation between success and hard-work is more immediate - you see that the people who work hard are more successful than you, and that those who slack-off are not.