I agree, the words "relatively little background" are too vague. What I had in mind was that the book requires little background relative to many other introductions to category theory (such as the grand-daddy of them all, Mac Lane's Categories for the Working Mathematician). But I should have been more specific. If update the arXiv submission, I'll fix that.
As cokernel points out, the level of knowledge assumed is roughly what you'd get from an undergraduate mathematics degree at an ordinary university in Britain (and probably many other countries too). I know this because I used it several times to teach a master's course at the University of Glasgow. Probably the most famous master's-level category theory course is the one that Cambridge runs in its Part III (master's) programme, which I've also taught. But this book covers much less than the Cambridge course, and assumes less background too.
If you've taken either (i) enough algebra that you're comfortable with rings, groups and vector spaces, or (ii) any kind of topology course, then you should be able to understand enough of the examples that you can get a good grip on the general concepts. If you haven't, then it might not be the right book for you. As others have pointed out, Lawvere and Schanuel's book Conceptual Mathematics assumes much less background than mine, and there are also texts oriented towards readers with a computer science background.