...Would you (or anyone else) say that improved sight reading is the key to playing more difficult pieces ? .......I'd love to be able to play the more challenging Chopin or Rach works but the only way to do that right now is to stumble and stop which I know isn't the best approach. I probably shouldn't even be attempting them at this point but its hard to resist !
Sight reading skills have nothing much to do with the ability to play "difficult" pieces. It is key more often than not however to the rate at which you learn a piece.
Excellent memorisers who might be weak at sight reader have a method to memorise their music quickly with too much reading. Memorisers need to read a small passage a few times carefully (determining which notes they do not know by ear) and slowly, then they understand how to always play that without reading it again. If a memorise needs to read they only do so to remind themeslves of the single note they might have forgotten.
A good memoriser has a close connection to their muscular memory, that is what it feels like to play a phrase of music, they have a much quicker aquisition of this than sight readers. They need to have this quick aquisition of muscular memory otherwise the repetitions to play a passage correctly would be impossible and they would have to rely on sighting the music.
Some memorisers can actually describe in words what they need to play. And its usually associated with patter observation like you say... Put 3rd fingers here and then you move to here which turns around to this chord and scale pattern which has this shape to my hands and then finish with this progression with this interval between these fingers. You describe what you need to do in general terms with actual physcial movements at the keyboard. and everything they do can be described in words. This is very helpful for memorisers to develop and indirectly will improve the sight reading.
A good sight reader however can use the sheet music to tell them how to repeat a passage. They slowly learn through many repetitions to slowly read less and less until you eventually memorise the passage. A good sight reader is has very strong conscious memory, that is statements which prompt us what to do at the keyboard. For example when you notice what a chromatic scale looks like by itself it doesn't become hard to notice it in the sheet music, thus all you must do is know the max and min point of the chromatic movement and all of a sudden you memorise many notes. Great sight readers see a huge amount of ways to see groups of notes at once, they can see chords, scales, arpeggio structures etc in all music.
Of course most musicans use a combination of sightreading and memory. They use sight reading to describe to them what needs to be done at the keyboard, they play what it asks for, they think about what it feels like in their hand (by looking at the hands, this takes our attention away from sighting and forces us to memory), they listen for any wrong notes (using sound memory), if there are any read again (using conscious memory to identify the part of the pattern you messed up).
Generally I find people cannot sight read a piece if they have little experience with movements that that piece asks for. It is of course good to learn new movements but there is always a matter of efficiency. I remember when I first discovered Scriabin Etudes/Preludes I had to learn them, but my teacher back then suggested to do them later on. I didn't listen of course and learnt my favorite ones, it would take me a month or more to learn one of them back then. In recent years I retruned to them with more knowledge about procedure at the keyboard and I can memorise them now on average in a matter of hours. It is just by magic if you work towards music how easy things become. I remember opening that book again thinking, oh I remember this, these peices took me more than a month to learn each, then I tried them again and it is all of a sudden much easier, I find I don't need to read every single dot anymore

You apply musical knowledge and you can absorb a lot with one observation.
So the more music you memorise the more building blocks of music you learn. Eventually everything becomes routine and there is just different directions to take. A good teacher pushes us to observe routine at the keyboard and how to apply musical knowledge instead of looking at every piece as an individual new lesson. Of course it is only after many years of practice that you start to understand ALL the types of routine at the keyboard. Personally I find Bach brilliant for sight reading skills, his WTC is my daily bread.