My father is probably the most logical person you can find. He is practically not attracted to music in any degree.
I might add to this with my own experience. I've been very logical my entire life. I've been heavily into computer programming, mathematics, and physics. And like DS's father I was never really attracted to music much for the bulk of my life.
In my teens I was attracted to rock & roll, but I think that was mostly just the beat of the music that drew me in. Plus maybe some of the lyrics and the romantic implications of the songs.
As I grew older I fell away from music altogether save for playing a little bit of classical guitar as a pastime. I never even listened to radio music. Not even in the car driving back and forth to work.
When I finally did become interested in music, I think it was a mathematical appeal. Partly to the sheet music, but also to the "mechanical" aspects of playing the instruments (piano, violin and guitar). I found classical music appealing because of it's mechanical or mathematical allure. Although everyone may not see that in the music.
But then Ironically, after I started playing the instruments then I began to become interested in the music just for the music. I moved from Baroque period stuff to becoming interested in more romantic stuff like Debussy's works.
So I think I came to the actually musical aspect of music from the long way around. Also, now that I'm learning to play Debussy's pieces I no longer see it from a mathematical vantage point. I now just see it as music.
I think there can be a mathematical appeal to written score, and possible the art of composing, as well as in the technical art of actually producing music on an instrument. Although, just because that mathematical aspect is there, I don't think that's what attracts most musicians to music. I think most musicians are just attracted to the music itself for the sake of the music. In fact, those musicians are probably the better ones as far as pure musicality is concerned.
Although, I think Debussy himself used some mathematics in his music. Wasn't he the one that wrote a piece based on the ratio of the golden mean? I don't think he used mathematics in all his pieces though.