However, if we were to suggest we are taking the E major chord from the E major key itself or the IV from the B major key would we able be able to temporarily change tonal center to the E or B keys respectfully?
Yes, you could change the tonal centre to either E or B major.
Example chord progression, passing through E major:
D, B7, E, E7, A, D.
Start at D major. Shift to E major using a B7 chord, only requiring one accidental of D#, using V7/II. Take that E major chord adding the 7th to create an E7 chord easily leading to V7/V. Once at A major chord, easily lead back to D major.
Example chord progression, passing through B major:
D, F#7, B, E, C#m, F#, B, E/B, A, D
Start at D major. Move to B major via F#7 chord using V/VI. E major chord is encountered within a IV-ii-V-I progression in B major. Use the note B as a pivot tone to create an E/B chord (now shifting to A major), rule of the octave voice leading towards A major chord. A major leads back to D major.
Can also lead back using Augmented 6th chords:
D, F#7, B, E, C#m, F#, B, Bb7b5, A, D
German 6th with cadential 64:
D, F#7, B, E, C#m, F#, B, Bb7, Dm/A, A, D
Smoother voice leading at the beginning, with Italian 6th leading back to D:
D, A7/E, D/F#, F#7, B, E, C#m, F#, B, Bb7(no 5th), A, D