Hmm... I think I should change my vote to melody. I've been thinking a lot lately about what I'm really responding to in music....and I think it must be melody, because i'm listening to a Sitar piece right now, which has ZERO harmonic movement (unless you consider mode changes a sort of harmonic change....if it is its too slow to be compared at all to Western harmonies), but the melodies and rhythmic energy of the piece are just wonderful! I could listen to this stuff all day.
On the other hand, the first prelude from WTC could be said to lack melody or rhythm, its just an arpeggiated sequence of chords. I love that piece....so...
I should have put an all of the above.