I cannot suggest why so many seem to harbour the angst you describe, maybe I am simply intellectually shallow and do not feel these things. On the other hand, I think I would far sooner work in the garden than spend my time struggling with music which gave me no joy or perception of beauty.
I don't think that music has goals. It is. The musician may have goals or reasons for playing; the listener may have reasons for listening. I used to regret that I was not given the opportunity for lessons, and had to go that route late in life, "too late" in some sense. But it also didn't get distorted for me. I played in the way a small child starts making castles in the sand, and gets engrossed. When sheet music was passed on to me when I was young, I started discovering what was in it; drew out of it what the composer had to say; discovered patterns in that which made me want to be creative. Music can be a personal thing. Yes, there are teachers, and there are people who "judge" what you do, but when the music belongs to you, then these are just foreign things to put up with. Something like that.