When I first started studying with my teacher, there were things I could hear and things I could not hear. What I could hear also was shaped by the context of my own playing, singing, and listening history for a lifetime. This is normal for everyone.
Some things to consider is that we can hear things in different contexts, and we can learn to expand those contexts. For example, I readily recognize V7-I, but my hearing was so "functional" and "intervalic" that in a way, G7-C, F7-Bb, B7-E, in a sense they were all "the same" - that is, my ear was attuned to quality and cadence, but not to specific pitch. Also, my ear was attuned to intervals more than chords, and melodically more harmonically (horizontal vs. vertical). Thus initially I might mistake Bdim for Bm because my ear hard the m3 but not the jarring tritone.
For hearing intervals themselves, I like the teaching idea of starting with large contrast first, for persons whose ear is not developed at all yet. m2 and M7 contrasts greatly with P5, M3, P4. One may hear "ugly-grating" versus "smooth-pretty" before distinguishing individual intervals (again, played together harmonically). Yet theory tends to start with P5, P4, M3 and m3, which is like distinguishing orange vs. pink vs. red, rather than blue vs. red.
My experience with chords is similar to yours, maybe. I could distinguish major and minor chords easily. The "dom7" was also rather instant as sound -- not sure if that still holds true in inversions. For augmented and dim7, my first breakthrough was to hear them both as "uneasy, strange, other" and decided to be satisfied with that as a first step. After a while, they were different from each other and this was a personal sensation --- possibly I hear dim7 as gentle-unsettled, and aug as blaring-unsettled.
I'm starting to learn to hear some other things. For example, BDFA is a "half diminished" chord or Bm7 with the 5 lowered making it (Bm7b5) as alternate name, but with D as the lowest note (DFAB) is it an inverted half diminished, or might it be heard as a "6 chord" (Dm6) because (some people's ears) my have the Dm sound jump out at them. The more important thing seems to have been to just keep experimenting, and listening from all kinds of angles.