I played this concerto a few years ago and still keep it up for performance (even though I haven't performed it in ages). Technically speaking, you're right, the concerto is not that difficult. In Russia, for example, it is considered an intermediate-level work.
The first movement is quite easy although the cadenzas pose some difficulties. The second movement is probably the most difficult (the middle section can get away from you very easily). The third movement (provided your thirds are up to par) is not that difficult at all (especially since the nerves are gone by this point). Just a caution: don't try to pull a Horowitz on the octaves unless you honestly do have the technique...they'll sound muddy, unvoiced, and, quite simply, bad.
This is only my opinion, but I sincerely feel that the difficulty level of this concerto is often exaggerated.
Topic: Tchaikovsky piano concerto - your opinion please