You should read along while playing, when you have memorised a piece already. It isn't that easy at first but sheet music is great for adding comments, ideas, highlight areas that you have to practise more etc. Make it your practise worksheet. With enough practise, reading becomes mostly automatic and all the extra information on the sheet music doesn't need to be stored in your brain, and the brain can concentrate on performance.
I think one reason for your good memorising ability is the fact that you can't read sheet music well, because you have to memorise quickly. But don't worry, it won't decay as long as you memorise music at a later step. You have to get free from your notes and the sheet music eventually.