I have a violinist friend (plays in an orchestra) and he says the hardest violin concerti are:
(in no order)
- Brahms
- Brahms Double Concerto, Violin Part
- Tschaikowsky (he says it's too overrated in terms of difficulty though)
- Paganini (pretty much all of them)
- Sibelius
- Sarasate: Symphonie Espagnole
- Sarasate: Zigeunerweisen
- Lots of relatively unknown Spanish music that is actually harder than the Brahms VC
In his opinion the hardest concerti are the Brahms and Paganini No. 1.
EDIT: For other instruments...
- Timpani: First movement of Brahms PC No. 1, Rossini's Guillaume Tell (entire opera)
- Flute: Tschaikowsky's Manfred Symphony, Rachmaninoff PC No. 3 finale, Beethoven Symphony No. 9
- Double Bass: Beethoven: Symphony No. 7
- Voice: Mendelssohn's A Midsummer Night's Dream, Rossini's Guillaume Tell, Mahler's Symphony No. 8
Obviously there are heaps more, but that's what I can remember from reading.