It was difficult in the sense that it's very rare (but by no means unprecedented) for someone to do it, but it was not particularly difficult in the context of my background. In other words, I'd say it was more difficult to become the sort of candidate admission boards would consider making an exception for than it was to actually get them to make the exception.
Not having an undergraduate degree is virtually always a nonstarter for graduate admission committees, which means you need to bypass them. The only way you can realistically do this is by proving your ability to an influential professor who has the clout to overrule them, or at least make them seriously consider it. That means you'll be trying this at a research university, and (ironically) it also means that more prestigious universities will consider it, though they'd never advertise it of course.
Before I accepted an offer, I was well into the admission process at both Oxford and CMU (the latter of which invited me to apply), and both explicitly clarified that my background wouldn't be held against me.
Specifically, I applied by 1) appealing to specific professors at the universities I was interested in whose research I respected; 2) explaining my unorthodox background with forthright honesty, while asserting the context that would clarify skipping undergraduate as a sensible decision; 3) getting very strong reference letters from reputable clients and past coworkers of mine; and 4) demonstrating through the interview process and personal letter that I had developed an exceptional skillset as an autodidact.
I'm very happy with the route I took. It enabled me to earn far more much earlier than I otherwise could have and without any debt; it also allowed me to accomplish a very specific goal: to study a specialization at the graduate level and contribute original research without needing to work through courses I was uninterested in. That said, I do recognize this isn't really possible for most people.