There's a whole lot going on here, so I'm going to unpack this in a few different points.
1. Anytime someone (especially a politician) suggests something both incendiary and unlikely (putting a domestic organization on the foreign terrorism org list), the goal is the same as a stage magician making a joke - it's to distract you. The correct question then to ask is what are you being distracted from.
2. It's hard to say Antifa isn't involved when they stated they are attending. As you say, they are not a monolithic organization but neither are white supremacists or most street gangs - we don't have a problem holding them accountable. Antifa does have a specific ideology based on pre-emptive violence, and they do have ideological founders and a history of violence.
3. Regardless of who you think may be responsible, this is a good point to realize that it's not smart to be associated with violent groups. They don't have the monopoly on use of force (that would be the state[^1] by definition) and therefore they'll eventually attempt to use violence, get noticed and get dismantled one way or another. Usually that process isn't peaceful when both sides are committed to violence, and there's a reason the state tends to win.
As I often point out with software, theory is great but reality tends to be something else. When your claim to fame is that you show up and commit acts of violence against people you don't like, well, you've sort of signed up to be blamed when you attend a rally and acts of violence start happening.
[^1]: The state here should be taken to mean the organized government as a singular entity