IRC has a number of easily abused aspects of the protocol, such as giving away the recipient's direct address which may give away a great deal about their physical location. Just like XMPP it makes the mistake of tying the user's presence directly to their TCP connection, meaning a connectivity issue (such as another user flooding their IP) results in them being unable to use the system. Outside of privileged bots on open-source oriented networks, there are typically no accounts or registrations for names, so network attacks are fairly common in an attempt to regain a channel or nickname.
Servers in a network generally have little to no proper recovery from partitions/splits, and a split may result in changes in moderator/administrative users due to the lack of persistent permissions. There is no attempt to recover discussions during a split, or to apply consistent ordering to communication. Splits, which should be transparent, become a highly impactful event to the users within a channel.
Due to the lack of packetized responses and the simplistic method of determining an active TCP connection, it is possible to (for instance) be disconnected because of asking for a list of channels on a slow connection.
It has a _lot_ of problems, and most of the ones I listed are at the network protocol layer. It is still usable for environments where there are very active administrators and few to no bad actors, but it falls apart quickly on the public internet.