Yea - but it should still be possible to structure such a compact in a way where you wouldn't create loser states - let's say you target 80% buy-in of states (by representative numbers) before activating and you're talking about the house (the same rule goes for the senate but the activation might fire at a different time) - The only scenario that would really flub the system is if the 20% of house seats not covered by the compact went all to one party (say the purple party) in close races while the green party and yellow party split the vote otherwise - in that case maybe the purple party won 11% of the popular vote and thus is over-represented by the 20% of control they have. Still the other states could balance the math to split the remaining 80% appropriately.
Additionally, as long as that proportion is significantly high enough (say above the 66% mark) the two parties that were partially disenfranchised would still be left sharing a strong majority in the house between them and, if something similar happened with the senate, they could coalition up and force through a congressional amendment that would bring the remaining portion of the country in line for the next election.
The Interstate Voting Compact couldn't be precisely similarly applied to congressional elections, it'd need more momentum behind it, but there is a way for the math to work out.
All that said, IANAL and I have no idea if the legal shenanigans used to make the IVC binding could also be utilized for a similar movement around the house and senate.