I'm surprised when I hear Prokofiev 2 put in the same category as Brahms 2 and Rach 3.
The only exceptionally difficult sections of the Rach 3 take place in the 1st movement, up to the cadenza. The original cadenza itself is quite difficult, but the ossia cadenza (which is typically played) is definitely not in the top echelon of difficulty, with respect to the standard repertoire. The Rachmaninov would be much more difficult if the tough passages were not constantly being alleviated by easy ones every minute or two; the primary difficulty in the Rach is just the sheer exhaustion one endures during the 1st movement. There are also a couple of sincerely nasty bits in the 2nd movement, but the 3rd movement is by in large not especially difficult at all. Honestly, the 2nd concerto is much more "tricky" and unpianistic, but it is just not as mammoth and physically taxing as the 3rd. The Rachmaninov takes much more time to learn or relearn than the Prokofiev 2, at least for me, but it just isn't as bad as its reputation. The Prokofiev 2's 1st and 4th movements are both complete disaster zones: The orchestra gives almost no coverage to the pianist throughout the whole piece, and the toughest sections of the Prokofiev are definitely nastier than the toughest sections in the Rach, albeit fewer and farther between. However, performances of the Prokofiev are much, much more likely to have a serious mess-up in a few places than the Rachmaninov due to this. It is far more "risky" than the Rach 3 to perform.
But as I quasi-mentioned, neither have anything on the Bartok 2. Bartok 2 is definitely the most difficult concerto in the standard repertoire, along with the Ginastera No. 1, due to the 4th movement. The 2nd/3rd movements of the Ginastera are very, very playable, though. However, I think that performance standards for the 4th movement of the Ginastera are extremely lax, comparatively, hence why I didn't mention it.