Earlier quoted context omitted.
> 1. Your private IP addresses are accessible by everyone in the same datacenter. This is pretty bad compared to an AWS VPC -- you basically have to manage and sync your own iptables between all your nodes.
Perhaps, but at the same time, you shouldn't be using IP addresses as a security mechanism. Assume the connection between your hosts is compromised, and code accordingly, with encrypted/authenticated connections between hosts.
If you're doing that defense in depth on a physical network, I'm impressed by your dedication but would avoid your work for wasting resources.