Which public key algorithm? In what mode of operation?
What are you going to use to actually encrypt messages? You don't want to directly use the public key primitives to do this.
In what mode of operation are you going to use that second, bulk encryption algorithm?
How are you going to authenticate messages?
What will you do to validate the public keys of your peers? When you close the application, will it forget everyone's keys? How do you prevent MITM on first contact?
What happens when your peers change devices, and thus public keys? How do you authenticate those changes? If you get any of this wrong, remote attackers can MITM messages.
How will you handle file transfers (and images and videos and voice, which will probably need yet another cryptosystem)? How will you cryptographically bind those transactions to the (presumably, somehow) authenticated chat session you set up?
What happens when someone's device is compromised? Is every chat they've ever sent also compromised?
What happens if someone is briefly compromised? Is every message they send in the future also necessarily compromised?
How will you handle updating your software when, inevitably, someone finds a vulnerability in it? What happens if you have to upgrade the whole protocol?
None of this is easy. Most of these problems by themselves are hard in their own right, but there's a combinatorics to them as well.