That isn't true. Silence is simply the absence of sound, and we as human beings are attracted to sound. You might say the same about a completely white canvas in an art studio - but the problem is, there would be nothing to pay attention to... it's blank.
Same applies to sound - silence is boring. Sure, it might be serene and a little silence is a welcome change, but too much silence would drive us nuts.
While I think you are basically correct, I don’t think it necessarily has to be that way. Silence itself may not be interesting but the psychological effect it has on us can be very interesting. And while it might not be possible to listen to silence per se, it really is possible to notice it and pay attention to it. But, of course, yes, too much of anything will probably drive you nuts.
Also, true silence is impossible anyway. There will always be something to listen to, even if it’s the sound of your own breath. Pay close enough attention to that and you might just notice something interesting about it. You might also start to notice all the other sounds you couldn’t hear before because of all the other noises around.