Mandarin is the easiest of any of the Chinese languages. It only has four tones instead of seven or more, there are a wealth of language teaching materials and a lot of natively-produced media, ranging from children's books to near-hollywood scale move productions.
Cantonese, on the other hand, has a decent amount of media but relatively little in terms of materials aimed at teaching foreigners.
Taiwanese learners have a decent amount of local television and music to lean on if/once they reach a high level of competence, but there isn't even a standardized written language! Instead there are the following horrible options:
1) Use an essentially unworkable system of trying to apply patchy etymological research of Taiwanese going back to middle Chinese to guess which characters modern syllable could be represented by.
2)Picking characters with similar Mandarin pronunciations, using a romanized script. This has the drawback that Taiwanese has several sounds and tones that Mandarin doesn't.
3) Use a well developed and self-consistent romanization (aka our alphabet) system developed by missionaries over a century ago. This is pretty decent but there are competing romanizations.
4) Use a mishmash of characters and roman letters together. This is very common.
One advantage both Cantonese and Taiwanese hold over smaller languages is that it's not too hard to find speaker communities in just about any major world city and many of those speakers are fully fluent only in that one language and find Mandarin to be a painful alternative. If you want to study an American Indian language like Blackfoot, you'll have the additional problem that every speaker of it is bilingual in English.
All in all, I don't think Mandarin ranks anywhere near the most difficult option for English speakers. It's certainly harder than romance languages, but in the big scheme of things it's of medium difficulty at worst.