Excellent call on the fountain pen! Rollar balls are nice as well, though the fountain pen really says something. That would make a wonderful gift. Make sure you give them some ink cartridges if you do that, or see that it has a convertor and a bottle of ink. Check out levenger.com then click on the link "pen and ink" to see lots of neat fountain pens. The Acme brand has lots of neat fountain pens priced below $100 US. Levenger "True Writer" fountain pens are affordable, and nice, though the ring-trim on the cap keeps coming off on mine. Namiki fountain pens are really neat!! The nib disappears inside the pen. Waterman, as mentioned, has a nice range of fountain/roller ball pens to choose from at all price levels. I personally love the simple and elegant Visconti pens, but they are more costly.
In college, I gave a teacher a good-bye gift of a certificate to the opera house in Milwaukee to see a production of their choice (because I knew they liked operas). It was well received though they've nothing to remind them of me, which is okay. I was happy they had fun at the opera.
When I was in third grade and changed from my first piano teacher to work with a college professor who taught out of her house, I got the first teacher a (yes..I know...ick, yuck) knick-knack though it was artistic and not just stupid. It was a 14" high sand castle I found at a Hallmark shop -- sand mixed with resin -- which had a verse on the bottom about dreams being possible, and a little hand written card from me that told her she helped my dreams come true. Knick-knack, yes, but personal.
Music teachers always have to watch the time so a nice watch engraved on the back would be something very memorable. Watches don't have to be super expensive, and engraving doesn't cost much at all. A famous quote about "time well spent" would be appropriate and fun.
Some of my students who have moved to other states have given me a little photo album (4"x5" things you can find at any card shop) with a few pictures of them in it. Inevitably, parents take pictures of their kids standing next to the music teacher at recitals or holidays and I almost always get those back with a personal note on the back. I keep all of these in the various little photo albums I've gotten, and they all reside within my piano bench.
This is a very inexpensive thing to do but touched me. I can go back and look at the photos. I don't ever keep those photos in frames however. I don't want one student to see a framed picture of another student and feel slighted in any way.
Hope this gives you a few ideas.
Topic: Good-bye present for pianoteachers