If you focus on this at the point where you aren't getting what you want its too late. You need to build a relationship of trust before you ask for an exception or special request (in the eye of the manager not you). Your manager should know the following: - That if there is a problem you will always come to him/her first not try and work around him - That he can count on you to do what needs to be done. You will go…
True to the extent that it is actually related to your job in a larger sense and doing so won't adversely affect your ability to do your 'real' job. Many managers will just keeping adding more and more responsibilities to their 'best' workers until they either start saying "no", quit or burn out, often without any malice or ill intent.
That being said there are better ways of phrasing it than "this isn't my job". Saying something along the lines of "if I do that I won't have time to finish X by next Tuesday as I promised. It might be worth checking with Dave and his team if they have someone who can work on it. Otherwise I'll be happy to do it, as long as you're OK with pushing X back a week".