I can think of 3 or 4 different reasons for this problem, and the strategies for coping might be different.
To me one of the most likely causes might be a timbre problem.
Let me tell a story, longwinded and peripheral as most of mine are.
We were in a small Lutheran church with no organist that Sunday, and the pastor wanted us to sing an unfamiliar hymn. He turned on the organ to plunk out the melody and hit the keys. Of course nothing happened; I sighed, jumped up, and pulled enough stops to make some noise.
He played the melody in the correct key and turned the organ off.
Then he started singing it IN A DIFFERENT and unsingable key, way too high for the congregation.
I remarked to my middle school age daughter, an excellent singer, that I didn't understand how it was even possible to make a mistake like that, and she said, "Daddy, I couldn't do it either. I can't match pitch to the organ, it sounds too different."
She would never make a mistake matching pitch to another voice. But the further you got from voice timbre, say an oboe, temple block, violin maybe, the more trouble she had.
The reason was simply lack of experience, and as she's grown up she's been able to add more timbres that make sense. It hadn't occurred to me then that this would be a problem, but I've played in ensembles all my life hearing pitches from different instruments.
The brass teacher Reinhardt advised learning to recognize pitches first from your own instrument, then from similar ones, then from different ones. A trumpet player would add trombone, then maybe clarinet, cello, etc.
Middle C on the piano has a distinctly different timbre than C a couple octaves up. It also has a distinctly different timbre than middle C on saxophone or tuba. Recognizing these middle Cs as the same note is a learned skill.
If this is part of the problem, then it seems to be trainable. Get a keyboard with multiple voices, or download some midi files, and practice singing back short (2 - 4 bars maximum) phrases.