Well, I was talking with one of the psychologists at work and she pointed out that I clearly have some type of NLD (non-verbal learning disorder). Never learned left-right, have trouble with number recognition and simple calculations even though I studied physics and math at the university level, did not learn simple tasks like tying my shoe-laces as a child and have difficulty with spatial recognition and kinesthetic processing. So not learning sight-reading in a normal way is just logical. I'll just let it be and hope things will gradually stick into my head or I find new ways of learning, I'm usually pretty good in solving problems my own way 
perhaps you have the learning disorder called dyscalculia. Here are some symptoms you may have shown as a child...
Frequent difficulties with arithmetic
Difficulty with everyday tasks like reading analog clocks
Inability to comprehend financial planning or budgeting, sometimes even at a basic level; for example, estimating the cost of the items in a shopping basket or balancing a checkbook
Difficulty with multiplication-tables, and subtraction-tables, addition tables, division tables, mental arithmetic, etc.
Difficulty with conceptualizing time and judging the passing of time. May be chronically late or early
Particularly problems with differentiating between left and right
Inability to visualize mentally
Difficulty reading musical notationMight do exceptionally well in a writing-related field — authors and journalists are more likely than average to have the disorder
Difficulty navigating or mentally "turning" the map to face the current direction rather than the common North=Top usage
Having particular difficulty mentally estimating the measurement of an object or distance (e.g., whether something is 10 or 20 feet (3 or 6 metres) away).
Often unable to grasp and remember mathematical concepts, rules, formulae, and sequences
Inability to concentrate on mentally intensive tasks
Low latent inhibition, i.e., over-sensitivity to noise, smell, light and the inability to tune out, filtering unwanted information or impressions. Might have a well-developed sense of imagination due to this (possibly as cognitive compensation to mathematical-numeric deficits)
Mistaken recollection of names. Poor name/face retrieval. May substitute names beginning with same letter
https://dyscalculiaforum.com/viewpage.php?page_id=1