...the core of music is structured rhythm and pitch relations.
May I add a thought? That the core of music is also in instrumental timbre? Some things simply make an impact by being played on a particular instrument. I once felt that the Cavatina from Deer Hunter made sense only being played on a guitar. On a piano for example it lost all it's beauty. This is of course something devotees of period-instrument performances will hold on to, but I wasn't thinking in those terms. But where does one draw the line? Bach's Art of Fugue is supposedly just what you describe, purely structured rhythm and pitch relations, a purely theoretical composition, not meant for any particular instrument.