You're probably sitting in the back with the percussion. Distance starts to really matter. Sounds travels slower than light. The larger the distance, the more that comes into play.
Watch the conductor. Don't listen to everyone else as much. Use the light as your cue to come in, not so much the sound of those around you.
Then play toward the front of the beat. You'll see (hear? or become aware) the lower pitched instruments in the back doing this. Your sound also has to travel from you up to the conductor, so if you play ever so slightly ahead of the beat, it can line up with the people in the front of the group.
The more you think about it, the more you're aware of it, the more it will really start messing with your mind for perception of sound, time, light, etc.
Rereading... If everyone else is coming in late though... Hm. Maybe they're not doing this. I was assuming the group was good and you were on the low end. Maybe it's the other way around. In that case, I'd stick with whatever the group is doing. Blend with them. It's up to the conductor to control the whole group, so if everyone's a little off, it's their fault, their responsibility to fix (or not). Some don't care, don't realize it, don't know how to fix it. Some don't bother with it and expect the group to fix that themselves though too.