Well, this is getting interesting. I don't think there's any way to prove an answer to the original question, that being whether it's easier to start on piano and move to other instruments or start on another and move to piano. Because no one has done it both ways! And we're also talking about what's the hardest thing overall, which I think is also impossible to prove, because something that's hard for me on a given instrument might be a cinch for someone else, and vice versa.
I play a lot of instruments, as do many other people on this forum. I did learn piano first and see it as a great starting point for learning general musical principles, greatly because of the visual aspect of seeing the intervals and chords right in front of you, not just on paper or in fingerings, but also of course because of experiencing harmonies and reading more than one clef etc.
I heartily agree with those who've said that it's difficult to be REALLY good at any instrument, and the possibility of true mastery is always debatable. The difficulties and outright obstacles particular to an instrument that one doesn't play may be imperceptible to the outsider, leading to assumptions like, "it's only one note at a time, how hard could it be," or the opposite, that fast scales are easiest on piano-- when, on woodwinds, you just lift or drop fingers onto keys that are lighter weight than piano keys and you don't have to shift position every three to five notes, so the potential speed is incredible.
Regardless of a person's starting instrument, there is also a factor about general learning that comes into play, especially if several instruments are taken up over time. It comes across to me like the learning of languages. Many people speak two or three languages pretty well (I'm supposed to be one of them, but I'm not). But when you talk to people who speak even more languages, and fluently, you start to hear statements like, "The first three are the hardest," which is kind of funny but also probably true, because the same thing is true of instruments (and maybe other things, but I'm not good at other things, so I wouldn't know). After a certain amount (and type) of experience, you learn how to learn certain things, and then you just, well, go and learn it.
Well, that's my two cents' worth.