Most people I know who like modern stuff have had go through a period of not liking it, and learned to see the beauty. That's part of the reward. In some ways it's like magic eye puzzles.
However when people make comments like 'it is awful and just random notes,' I see that as an embarrassing statement by them. I always happily admitted (pre-liking modern stuff) that I was ignorant, rather than blame the composers.
You're forgetting certain option there
You're saying that for one if you like it you must have seriously thought and pondered about it and listened to it, that if you don't like it you must surely have never heard it or seriously thought about it, that if you don't like it must be because you're ignorant and it's not the composer fault.
You're forgetting the people who have seriously listened to it and researched and analyzed the philosophy this music comes from, more thoroughly than many people who are fan of this genre. You're also forgetting that you must not be ignorant at all but the music just doesn't strike any chord of any kind intellectual or instinctive for you.
It's not a crime not to like something and I find it very hypocritical to claim that must always be the fault of the listener. I'm not claiming that it's the fault of the author either but just that there's must be a sinergy between the author and the listener, and often there's none and it's no one fault.
Besides whatever piece of art which needs someone to have studied and understood the technique and compositive means behind it in order to be able to appreciate it, has already failed completely. Art is about using complex techniques to express universal content that everyone can relate to, and the "secrets" of the techniques and the means are something that only the artist should know. No serious art expect the audience to have a knowledge about the techniques used to create it. To expect someone to be able to appreciate an artistic work only after having studied a giant book explaining it and its technical characteristics is just ridicolous and anti-artistical by it very nature.
There's a group online of people who don't particularly appreciate modernist music and just fight for their right to compose whatever they want and have it accepted as worthy by the accademical world imprisoned in its ivory tower away from the real world and anything removely living.
These people are composers, musicologists, performers, orchestra members, teachers, musical theorists, music reviewers, philosophers, artists, singers in other words people who are neither young and dismissive of the unknown and neither ignorant about this style and genre of music (because no mistake, this music is not the universal future, is not what music in the modern time naturally is and is supposed to be, this music is just one of the hundreds of genres and styles that exist out there, with its own characteristics and motivations ... like all the other styles and genres)