With Elliott Carter, Birtwhistle, Ferneyhough I feel that their names get included more from piety than affection, but I'd be happy to be turned around on them.
Is there no interest on these forums in Hans Werner Henze?
I like the small number of works I've heard by Henze and would certainly like to check out more of his output.
I don't remember my top five on this thread, though I'm certain it included Carter, Wuorinen, Norgard, and perhaps Dusapin and/or Dutilleux. Recently, Poul Ruders might find his way there as well. Of those, Carter was probably the most difficult one for me to come around to, and I'd be lying if I said that I liked the whole of his output without question.
A great help in coming around to my favorites was the prolific amount of recording that's available from those composers. Though many of Wuorinen's works (and especially his recent pieces) have not made it to CD, the amount of older pieces available is pretty impressive. The same goes for Carter.
I would agree that it's hard to level judgment about modern composers whose work hasn't made it to record, or whose recordings are excruciatingly difficult to track down. Lots of recent composers would fall into this category for me, including Ib Norholm, Allen Anderson, Martin Boykan, Rand Steiger, etc... Many of these composers could likely be favorites of mine if I knew more about their output.
I would also have to agree with you about Ferneyhough's appeal. 80% of people I've encountered who celebrate his genius seem to feel better about weaponizing him as a heavy namedrop than about how any of his works affect them. Maybe the same goes for Carter, though...
After hearing his excellent Naxos quartets, I'm a little bit tempted to include Peter Maxwell Davies, but I don't know much about his other works.