Suppose the amount in the envelope you choose is $100. If your amount is the smaller one, the other one has $200. If your amount is the larger one, the other one has $50. So the expectation supporting the switching argument is computed over two different scenarios, where the amounts in the envelopes are {50, 100} and {100, 200} respectively.
This calculation seems to violate a hidden logical constraint: the set of amounts in the two envelopes must be the same no matter what conditional branches we consider for the expectation.
Another related issue - if you switch, the argument for switching back assumes the amount in your original envelope is 4x what you originally said it was.