I think it would be unwise -- at best -- to say that there is a specific limit, as that implies that there is some sort of outside authority to set that limit. Which there really isn't.
However, there is an authority of sorts: a combination of the composer's intentions, as indicated in the score (tempo markings, dynamic markings -- sometimes comments, even the title!) and a knowledge of what the composer's intentions beyond that might have been, as indicated by the performing style and characteristics of the era in which the piece was written.
Thus the performer needs to not only know enough about playing to accurately read and interpret the markings in the score, but also she or he needs to know enough about the general characteristics of the music to be able to judge what the composer probably had in mind -- in other words, a firm grasp of music history and the music of the time and place.
This is not to say that one should attempt to slavishly interpret and reproduce what the composer meant. Sometimes that works; some of the early music groups do a marvelous job, for instance. Sometimes it doesn't -- it would be very hard to convince me that a slavishly accurate performance of the Beethoven concerto #5, Emperor, on a period instrument with a period band would be better than a performance on a modern concert grand and full orchestra (although it would be interesting...).
So then what I am saying is that there often is quite a bit of room for interpretation, and whether a given interpretation is "too far from how it's written" is to a great extent a matter of taste and current performing custom (which changes from time to time!).
I suppose the real bottom line on it is a two fold question: does my interpretation bring out what I think the composer had in mind, and does it say what I want to say with the piece?