Saint-Saens did indeed write five piano concerti: No.1 in D, No.2 in G minor, No.3 in E-flat, No.4 in C minor, and No.5 in F, the 'Egyptian' (my personal favourite). Nos.2 and 4 are perhaps more well-known than the other three, with No.5 running third. (I have a soft spot for No.3 but would have to acknowledge it is of lesser quality than 2, 4, and 5.)
As for the four listed, well, as my teacher used to say regarding the Rachmaninov 3 vs. Brahms 2 debate, they're all challenging in different ways, and which is more difficult depends on what gives you more difficulty. While I've studied the Beethoven, I haven't studied the other three (although I have listened to them numerous times), so I can't say much about their relative difficulties. The Beethoven is difficult if you have small hands or struggle with extended scale passages, but I do think its overall difficulty is sometimes overstated.