You are all right, in that a title does not have to limit your imagination when listening to a piece. In fact, I would encourage anybody who heard my music to think of whatever they want when they hear it.
But I also find it fascinating to hear a piece from the composer's point of view... for example, take The Rite of Spring (also in Fantasia). I first heard the piece without knowing the titles of the sections, and later when I found out it conflicted with what I thought. But the very fact it was in Fantasia shows that a piece of music can be interpreted many ways.
This debate between absolute and programmatic music has existed for over a hundred years..... I quote from the Harvard Dictionary of Music:
"The debate between the proponents of absolute and program music has continued for over a century, orelapping, but only in part, with a debate over the extent to which music is a form of expression. In the 20th century, the proponents of absolute music have generally had the upper hand; the influence of the doctrine can be seen in many workds of musical aesthetics, in Schenkerian and other approaches to musical analysis, and not least in the relatively objective, intellectual, or formalistic character of much recent music. Nonetheless, hte dichotomy between absolute and program music is essentially misleading, for it obscures the complex intertwining of extramusical associations and 'purely' musical substance that can be found even it pieces that bear no verbal clues whatever."