Earlier quoted context omitted.
At least with packages, there are "no-X" versions of most packages available (useful if you don't have X on a machine).
the problem isn't that it's dependant for X11 on packages.. moreso that the port system requires X11 to function.. http://comments.gmane.org/gmane.os.openbsd.ports/54692 http://www.openbsd.org/faq/faq15.html#NoFun (at the bottom of this section)
My point about packages still stands, though; I don't use ports on my OpenBSD servers (since it's much easier on server load to just download/install a package than to try to build things from source; well, that and the lack of a need to install a whole lot on an OpenBSD server besides a runtime for $SERVER_PROGRAMMING_LANGUAGE, but whatever). I suppose if you're particularly paranoid about downloading things from the internet, you could build packages on your personal workstation with the necessary env vars to disable X11 support in the package, then move those packages over to the server and install them there (basically, doing what OpenBSD's ports/packages maintainers already do in order to create the packages in the first place, but on your own).