I do pre-prepare, as you mentioned -- and precisely because using it can cause distraction (it's not so much the key movement distance, as just simply the movement for me).
That said, I use it to change tone colour, not to change volume (although it does decrease the maximum volume, obviously -- but you should be able to get PP without it, at least on a decent piano). I don't use it all that much.