I agree with everything lostinidlewonder says here, although semantics, as usual, can lead us to talk about one state while apparently meaning another. The act of spontaneous creation is a wonderfully ecstatic process, and produces the subjective feeling that we, as players, are observers, conduits, of an infinite stream of beautiful events coming from outside ourselves. Listening to what must amount to thousands of recordings I have made over a lifetime, I can confirm that this is the state which produces the "best" music - best in the sense that I would wish to repeatedly hear it and put it onto CDs. Curiously, it also leads to my completely forgetting what I have played immediately afterwards; again, very like a dream, which takes great effort to recall later in the day.
My teacher was supremely accomplished at the consciously calculated type of improvisation, which he could produce at any time, on demand, in almost any known style. Practitioners of this sort of improvisation often possess very strong natural musical gifts - perhaps absolute pitch, perfect short term memory, immense harmonic knowledge - that sort of thing. I found my teacher very intimidating for many years, as I could have worked at it for a century and never come close to doing the things I heard him do. But then, over the next couple of decades (I was a pathetically slow thinker - still am) I began to realise that, once the initial awe of skill had abated, his sounds rarely moved and transported me in the way my own recordings did. As lostinidlewonder says, it is frequently formulated and boring after the initial impression of brilliance subsides.
These days, working at the creative habit over five decades has more or less enabled me to enter the transported state very quickly, although even now, most of my recordings present, at least to my own ears, about ten minutes of "taxiing prior to take-off".
I have described my personal analogy of the chaotic feedback loop elsewhere so I shan't repeat it here, but until somebody tells me of a better theory I'll continue to nurture it as it serves my musical purpose.
Where I think good improvisers could direct more conscious effort is in discovering more processes of improvisational form. It seems to me that the forms of improvisation are necessarily dynamic and organic, whereas those of composition are architectural and static. It's like the difference between a fine cathedral and a living organism or plant. Both can be beautiful, the difference is not in beauty but in the mode of creation. You wouldn't breed roses to imitate cathedrals so why push improvisation into the static confines of sonatas and fugues, or for that matter rags, blues or sixteen bar phrases ?
Improvisers do not talk anywhere near enough about these things, and virtually never discuss them in everyday language listeners and beginners can understand. This is a pity. Preserving mystique might bolster egos but does precious little for the right of everybody to create in sound at an instrument. I think in this respect we are pretty lucky on this forum; the improvisers here are down to earth and helpful.
Topic: three different mental states for improvisation