Beyond the "You can write Fortran in any language" argument about properly structuring your code, there is a certain amount of truth in this. But in my opinion, this is mostly about familiarity. It's like a human language that uses another alphabet. Sure, if your native language is English, Polish looks more readable than Russian, as the latter is using a different alphabet. But that's a literally superficial point of view (cf. Lisp's parens), a stepping stone that is easily surmounted and doesn't change the total learning curve a lot.
English does look simpler than French, too. On the other hand, once you get beyond the accents, you know how to pronounce French, whereas that's not the case with English (cf. "ghoti"). Personally, I never was that bothered by e.g. the type characters, and quite often they were quite helpful about the context expected/required. But then again, I'm German, so maybe growing up with a somewhat ridiculous grammar and Funky Capitalization helped (there's a Perl module that allows you to write code in Latin, and here the cases and other grammatical structures replace the funky characters).
Perl certainly has its weaknesses. Some operations should probably be bound to types/objects instead of C-like functions, references are often difficult to untangle and the default object system ain't that grand (on the other hand: Moose). But I've yet to find a language that doesn't have similar weaknesses. Due to the somewhat funky syntax, they're just quite obvious. What does that dollar sign right there indicate? On what default variable is this code operating? Schwartzian what?
But I would call that "traceability" issues. And this certainly also happens with macros, meta-programming, complicated object hierarchies, generators, decorators, DoI containers, etc.
It probably does matter for casual programmers, i.e. people who don't work 20+ hours/week in Perl. Sysadmins come to mind. But if your involvement with the language is beyond that, I think "readability" is quite often a matter of taste, not more. Which, of course, doesn't totally disqualify this argument. It's just more a matter of rap vs. punk, not junk food vs. rucola salad.