Look at the score, and count the beats. You will see that the Brahms violin sonata isn't actually that slow.
Vivace also means lively, not fast. And the piece is rather lively, in a very innocent way. It's not exactly a Strauss waltz sort of lively.
I love this piece, it's amaizng.