In my opinion, it is okay. Even major composers have done this. (I dont have any citations but i know its happened) Once you build off of the thematic elements, it can often be made to sound completely original to the point where you cant tell.