Hi,
No 4 is my favorite PC in all concerto literature! I'm currently working on it for a performance in April. (I also played twice before, years ago.)
Whether you like 4 or 5 better is a matter of personal preference--No 4 is more introspective while 5 is full of bravado, so their overall appeal is different.
I think both are quite difficult, and possibly 4 slightly more so. It requires a great deal of control in the 1st movement, with rapid, yet extremely light and even passagework. There are a lot of leaps in 1st and 3rd movements. Most challenging, the interpretation--I have found that too much concentration on detail for too long results in a loss of forward flow of the music. The sheer beauty of the lyricism is stunning, but takes a lot of effort to produce the "effortless" quality! My goal is to have this quality without revealing to the audience that I have put literally hours into just the FIRST line.
Teresa