I just came across this thread and thought I'd throw my 2 cents in, even though it's a few months old, as I teach ear training for a living and I think I can be of some help!
Fluxit, it is definitely achievable to develop a very good sense of relative pitch in well under 4 years, with less than 60 minutes a day, regardless of age. I've worked with plenty of older students who have made great progress, so I think it's always achievable.
Maybe an important point to make is that, as you suggested, ear training (for most of us at least), never really ends. Your aural skills can always get better, so you can recognise and play faster/more complex music. Learning to play atonal music or very fast/complex tonal music by ear within 4 years would be a huge feat. But learning to play
most tonal music is definitely doable.
It looks as though the source of your problems is related to the approach you've been using (which is the same problem that most people have with ear training). Bob has hit the nail on the head. Using a tonality based approach that focuses on using scale degrees or solfege is much more effective than focusing on intervals. The tonic is a stable reference point that you can use to recognise any note or chord you hear. Intervals on their own don't give you enough information.
If you start focusing on learning to recognise and sing scale degrees/solfege, you'll see noticeable improvement quite quickly.
Regarding some of the comments about focused ear training vs incorporating it into other practice, I think they both have their place. Once your aural skills reach a certain level, you start to train your ears naturally whenever you play or listen to music, because you can keep up. Before that point (and after it as well for certain things) it can be useful to do some focused ear training, so you can slow things down and give your ears a chance to hear and work out the individual sounds that you're learning to recognise. Having said that, putting it in context and trying to work things out by ear is absolutely essential, and probably the most important part of training your ears.
I hope that helps a couple of people! If anyone's interested in more info about ear training you can check out my site:
https://www.eartraininghq.com (I hope no one minds a small plug

)