Whether or not it's "too late to start" probably depends on what your personal goals are and what you wish to to achieve through music education. The same can be said for choosing between classical and jazz; it's up to you to decide what you want and need from the program and what you ultimately find most satisfying.
Your concerns make sense, though, and I think your generalizations about the strengths, limitations and distinctions between classical and jazz approaches and practices are accurate. But any number of professional musicians manage to "cross over," and it's certainly possible to excel in both idioms if you have the interest and determination to do so.
If your school forces you to decide on one musical focus at the complete exclusion of the other, that's unfortunate.