In my opion physical movements the way you put it is something that we have to make in order to make music. It is the way we move and how good we move in order to make the notes from the score our music expression. On the other hand when you minimize them to the point that they can't be seen from us we can't blame the performer that he doesnt move quite enought in order to play in the way that i have mentioned above. It is the sound he/she produces that we'll judge.