However, if I really just let go, there is very little shape (as most people would see it) to my expression right now
What other people think is irrelevant. What you feel is everything.
It does not come out in arpeggios and chords that are built in proper manners.
Oh dear ! How sad ! Report to the musical headmaster's office for the strap. "Proper", "ought to" and "should" can be completely removed from improvisational considerations. "Do what thou wilt shall be the whole of the law" - Aleister Crowley.
I can sit down and decide that I would like to use this or that kind of chord progression (though I frankly have a very difficult time thinking in this manner) and so on, and I can "improvise" to a degree in this way, but it isn't all that satisfying, to be honest. It doesn't have the raw elements that I experience in my description above.
Of course it isn't satisfying. Of course you have difficulty thinking that way. Your reactions are quite normal for someone with orthodox training. What you do is start from a position of freedom and gradually, over a long time, allow form and order to enter the mixture. These latter are better if they are your own form and order, meaningful to m1469, not arbitrary sets of rules handed down by "experts" or a bunch of boring, dead Germans.
Loss of inhibition is the big hurdle for mature people starting to improvise. It helps to form the habit of taking pleasure in abstract sound for its own immediate sake. A flower is not beautiful because of the precise measurement of its petals or because the bush has a certain number of leaves. As Charles Ives said, "The stones in the old garden wall by the orchard are all different." Once the key to the eternal present is found, and becomes your possession, the rest will follow.
It does take time though, largely because it is a life-long, open-ended activity. It's totally different from learning pieces. You don't grind away at it day after day after the manner of a hard job like learning Mazeppa or painting the house. Once flow is acquired, you can pick it up and put it down as you wish. A "hard job" mentality will kill it before it starts.