Perhaps you should consider "technical difficulties" not necessarily as explosive displays of virtuosity (although certainly there is room for that as well) but as a way of creating an aural *effect* on the piano. It's mostly about what kind of sounds the pianist can make...all the orchestral instruments he can imitate. That usually involves advanced knowledge of physical technique, and in my opinion, composers were right to make their pieces as difficult as necessary to make them what they (mostly) were: masterpieces. Technique is often looked down upon, for some reason...but in fact "music" and "technique" need to remain very closely linked. To me, music involves the melody, phrasing, etc...and technique involves the means through which the music is expressed. For example, La Campanella wouldn't be quite the same if, in the beginning, only the "bottom" right-hand notes were played along with a bare left-hand accompaniment. What if the "trills" section were replaced with lazy single-octave jumps?
In some cases, yes, only technical display is present. Most of the time, however, the difficulty of any given piece serves a purpose. Simplicity of technique does not guarantee a masterpiece; nor does impossibly difficult writing. It is also important, in regards to the pieces' lengths, that the "omissible" sections you speak of typically serve as worthy transitions from sections to sections (if I am certain I know what you're talking about) and without them, the pieces would undoubtedly seem choppy.
Still, if you want to avoid pieces without high technical demands (and almost endless "music" as you say), look no further than the works from the Baroque and Classical eras--most of the great pieces of those ages have modest technical demands but take years to master, substance-wise.
Topic: Why does so much classical music lack quality melodies?