I prefer the IV (it is my favorite LvB piano concerto)
I have not the slightest idea of which is more difficult. Being at Clementi sonatinas level (i began playing less than 2 years ago), those works belong to another world for me...
Beethoven's 4th PC is one of the greatest concertos ever penned. The motivically integrated yet lyrical beauty of the 1st movement has no equal in the repertoire. A perfect gem. 5 is very great, but not quite as great as 4.
No 4 is my favorite PC in all concerto literature! I'm currently working on it for a performance in April. (I also played twice before, years ago.)
Whether you like 4 or 5 better is a matter of personal preference--No 4 is more introspective while 5 is full of bravado, so their overall appeal is different.
I think both are quite difficult, and possibly 4 slightly more so. It requires a great deal of control in the 1st movement, with rapid, yet extremely light and even passagework. There are a lot of leaps in 1st and 3rd movements. Most challenging, the interpretation--I have found that too much concentration on detail for too long results in a loss of forward flow of the music. The sheer beauty of the lyricism is stunning, but takes a lot of effort to produce the "effortless" quality! My goal is to have this quality without revealing to the audience that I have put literally hours into just the FIRST line.