However, many companies face similar issues. For a moment I thought the OP worked for my company because we face the same issues (that's not possible however because in person stand ups are banned at our company due to the distributed nature).
1) While I agree that 20 people seems like too large a team in general, I have a fundamental problem with the idea that scrum can determine what size team is right.
What seems to be happening, it appears to me, is that we have a fair idea of how large a meeting can be before it becomes unwieldy, and scrum has a daily meeting as it's fundamental component, so basically decides that for it's fundamental component to be effective there shouldn't be more than 8 people so therefore teams shouldn't be larger than 8 people.
That seems backwards to me.
2) Almost every scrum training usually has a separate scrum master. I do think the idea of a team member being the scrum master makes a lot of sense and would probably alleviate a lot of the concerns.
3) Scrum story points are supposed to reflect end user benefits. A refacorin's benefit will only show up after several sprints. In the meanwhile you hurt your velocity significantly because your refactoring is likely to be 0 points. Now, this actually makes sense to me, but the issue is that in most places management is carefully monitoring velocity and is gonna hold the drop against the team.
To be fair though, I think this is another issue where the real problem is management converting a measurement metric into a target, which basically negates any use it may have as a metric.
My biggest issues with scrum are:
1) A daily standup is ridiculous. It's highly disruptive and gives me the feeling of someone constantly peeking over my shoulder. A better strategy is a simple email when someone has an update, or a blocker to the rest of the team. Very few of my projects are done in a single day, and it's psychologically stressful to join every meeting saying I worked on story A, will continue working on story A for 3-4 days in a row, while the managerial types rattle off a list of meetings they attended, emails they sent, trainings they completed, etc.
2) Sprints are far too inflexible. Each story should basically define a new Sprint for the people involved in that story, which may or may not be longer than originally expected. I don't see the point of having a fixed 2 week period that applies to everyone. Why not just have a weekly or biweekly meeting where you go over everyone's stuff and see where you stand. It's basically a more flexible version of what sprints try to achieve, without the psychological issues created by having to fit things into a Sprint or alternatively having to carry it over into a whole new Sprint. That same meeting can track how many points were earned since the last meeting and you have your velocity being tracked as well.
Retrospectives are great. But they should be included in the same 2 week catch up meeting.
I find scrum tries to create a lot of artificial deadlines, possibly to encourage people to break things into smaller pieces and make regular progress, but I find that's not how people usually work. In my experience people tend to work more in spurts, delivering a ton of features and bug fixes over a couple of days, and then going relatively quiet for the next few days. Not because they aren't working, but because thats just how things usually pan out since software development is a decidedly non linear process.