What do I say to my pastor when I tell them I am moving on?

I think you should be honest with him. Completely honest. After all if he is hindering you in your ministry how many other people is he doing the same too. A minister is supposed to encourage us to do Gods work whatever it is. Ill pray for you as this is a tough situation. Pray before you go in and ask God to guide you and to put the right words into your mouth.