I've seen Agile at a handful of shops, a couple dozen projects. I've seen it implemented well just once...
The "scrum master" was a dedicated role, filled by a technically able person, who sometimes helped a little bit with the coding. This individual had read several books and taken a course on Agile. They were sincerely passionate about Agile and wanted to implement it effectively.
The "project manager", a DIFFERENT PERSON was part of the "business", and interfaced primarily with the scrum master, but they were with us for several hours of both planning and retro.
Our sprints were 2-3 weeks. At the end of the sprint, we spent AN ENTIRE DAY on retro. Before the next sprint started, we spent AN ENTIRE DAY on planning. We used a physical board which was matched by the tracker. They were kept in sync by individuals and validated by the DEDICATED DOCUMENTATION PERSON.
We broke down the tasks relentlessly, until there were almost no 3-point tasks, maybe one or two 5-point tasks, and everything else was 1-pointers. I think this one is the most bang for your buck, lowest hanging fruit, easiest to implement winner. Someone has to enforce it vigilantly until everyone is used to it.
Those three things are, IMNSHO, the keys to getting halfway decent Agile going.
Edit: Was one of the least soul-crushing and nicest experiences in professional software development. The planning meetings were a serious effort, but also a pleasure because I liked everyone on the team, and the large time windows made it not seem rushed, so we had time to joke around and shoot the shit.