Just practise sight-reading as much as you can, and as widely-varied music as you can find. Find cheap music from second-hand sources and buy it, and run through it - choose what interests you, which you feel you *must* explore, rather than what you think others might "expect" you to choose. Stretch yourself by finding music that may be too difficult for you, and that looks interesting (you want to make this enjoyable and interesting), and just sight-read it - in slow-motion if need be. Go back and run again through pieces that seem to interest you, so you gain at least some familiarity with them, even if you don't go on to learn those pieces properly.
It's difficult for me to judge what my own musical skills might be, and my opinion seems to change according to my mood. But an apparently excellent ability to sight-read music, and to cope with complexities like lots of accidentals, and so on, is one thing that other people do seem to consistently praise me for, so I can assume that that, at least, has some foundation in reality. And if it's so, I attribute it very largely to my practice over decades of finding lots of second-hand music in many styles and sight-reading it (not always going on to learn it properly).
And I never did this consciously to develop skills like sight-reading or becoming more fluent in notation, but just because so much music I found intrigued me and I wanted to find out how it was constructed - and it had that effect anyway. I always wanted to compose, and that in itself seems to give you a heightened urge to understand how other composers construct their music. And it's possible (although I'm not quite so sure) that composing itself helps make your notation-reading abilities more acute.
Regards, Michael.