I did what you are considering. Majored in Physics and Math but spent most of my time in the music school building. Major instrument was (and is) piano minor was trumpet. I am 65 years old.I found it way easier to make money as a scientist & engineer, went that way, and made a good life. But I never stopped my piano work.Now I find that piano ('keyboards' as I am into midi performance stuff) was the ONLY skill I ever had that never stopped growing. It is the unifying force in all my life.Also, as I became successful, I found that what I really wanted to do was get some time at the keyboard, after the day or week of business pressures.