How should I go about improving, or am I just hopeless?
No, no, no. Nobody is "hopeless". Dismiss that thought at once and don't let it return - ever. The main issue with personal improvisation is that the states of mind involved, the learning of it and the practice of it, run contrary to the ways we are conditioned to believe learning must take place. To improvise - and it is a lifelong, organic process, ever changing and developing, not a three week, three month or three year "course" which finishes - it is best to work from a position of freedom toward one of order. This goes against all orthodox, serial learning and conditioning.
If you start by trying to impose order and structure on yourself, trying to compare yourself with other music or players, before acquiring flow, you will never get off the ground. If you can just enjoy your sounds, let things flow in easy rhythm, without worrying about "rights", "wrongs", "shoulds" and "ought tos", then after doing it every day it has to develop over time; it cannot do anything else if you are musically sensitive.
Absolute pitch, or indeed, most traditionally measurable musical abilities, are unnecessary beyond a certain point in order to create in music. It sounds really silly, but it's true; the faculty is quite different.
I think birba is absolutely right about fingering in improvisation, and it is a very important point, because consciously thinking about fingering will impede flow, if not cripple it altogether. In performing pieces we search for special, optimal fingerings to suit special situations. Improvisation does not allow time for this perfection because of its spontaneity. Instead, we look for large subsets of general fingering, ways of fingering chunked sections which we get into our head and then relegate to the subconscious. While not always optimal in the microscopic physical sense, these chunked fingerings can be applied to musical effect instantly in any position (key, grip) on the keyboard to sound a spontaneous idea.
Again, it's a way of thinking quite foreign to traditional learning.