I disagree, I find it to be a very well constructed piece. Then again, I love listening to Mahler and Bruckner so perhaps I have different tastes as far as length. And it's not because it's hard, because I would bet money that Rach 3 and something like Scharwenka 4 are harder. With time it becomes apparent that the Busoni holds together well almost everywhere and is easily digestible. It stands above many other concertos in my mind because of the cerebral genius that is apparent throughout a lot of it, which I don't see in enough Romantic concertos. There are many virtuosic passages but they have a point and a sense of direction that keeps the music going forward. The entire fourth movement particularly emerges as constructed in one continual flash of inspiration - there is no way that Busoni could have come up with almost anything in that movement by sitting at the piano and stamping it out.
And pianistimo, I'm not sure what you're referring to when you say that the piano takes a long time to come in - it's about 3:35 by my count, which is about the same in a lot of classical concertos and in Brahms 1.
I have come to these opinions from listening only to the version played by Garrick Ohlsson, who grew up around the music of Busoni and knows how to carry it across. Perhaps Hamelin, in his slightly lamentable tradition of passing through the music of a composer so fast as to often fail to get a feel for that composer's style, does not bring across the concerto nearly as well. The three minutes I have heard of his version come across as plowing through the music way too fast.
Perhaps, cz, you might be able to find a video of Ohlsson playing the concerto with Dohnanyi or something, but I've never looked for that. Good luck in finding something.