Jazz is a specific style. Baroque is a specific style. Classical is a specific style. Romantic is a specific style. And so on. What classical pianists do is "spread themselves too thin," and learn how past composers' fingers moved from five or six different eras. It is a lot of work. So much work, it crowds out improvisation (in most cases). The reason jazz succeeds is that it is focused on a specific style, with a specific vocabulary of harmony, melody, rhythm and technique, so the task is simpler.
However, this does not mean improvisation cannot fit into classical. But we must be willing to either focus on a specific style or no style at all (in other words, find our own individual style), and possibly make some sacrifices on how broad our classical repertoire of pieces is. Until the overall attitude towards what being a "good pianist" is changes, I believe classical pianists will continue to "spread themselves too thin" with learning how dozens of different people moved their fingers, rather than learning how their own fingers can move in a new way.
Another way to put it is classical piano playing today is more or less a museum profession, it is not a creative profession. The point is to become experts at a wide variety of musical history, by a wide variety of people. The point is not to create something new. People always think "composition" when thinking about classical music, but this is only because people are accustomed to striving with great misery and discomfort to learn difficult ways of moving their fingers that are in many cases not natural.
To change the classical world back into a creative one, we must remove the artificial distinctions between styles, remove the extremely high performance standards (as the ONLY criterion for "good musician"), and remove the endlessly recycled vanity and pride amongst modern classical composers. I think it is already happening, it is only a matter of time before this becomes true.
Those were just some personal thoughts, from an amateur who thinks a lot in isolation. My thinking is influenced by my profession, software engineering. My entire job is based around battling complexity and making things simpler. So, I believe the reason improvisation stopped happening in classical is simply that the task of being a "classical pianist" became too demanding and too complex after a while.