The simplest answer is, if you don't enjoy it, don't do it. Find another hobby.
The longer answer might be some questions. Are you really sure you'll never be any good? I started when I was 40 (I'm now 65) and it was not until the last 9 years that I had a good teacher and only in the last 3-4 that I've felt confident at playing serious pieces, 7-8 Preludes and Fugues from WTC, Beethoven's, Opus 2 number 1, his Pastorale and "Funeral March" sonatas, a bunch of late Brahms, Janacek, etc. It's not Scarbo or La Campanella, but it's a bunch of interesting music that I can play for myself and my friends. It took a long time and a lot of work, and also it took finding a good teacher (or two).
The next question is ...Can you find a way to enjoy the process of making slow progress? Do you like working out technical things and finding a bunch of ways to practice particular difficulties? A teacher can help with that, and there are some good (and some bad) teachers on line whose videos may help you think of new approaches to practicing.
What is the long term goal? If you are really only interested in very virtuoso music and nothing short of that is fun for you to work on, then you may really be in a "can't get there from here," situation. But if there pieces, like the ones I mentioned, that you might like to play, and there are intermediate things to learn along the way that you can enjoy, then it may take some time, but you might not really be in a "never going to be able to be good at it," position.