I was not a music major as an undergrad, but took several music elective courses including conducting. The final exam was conducting the university symphony orchestra, which had over 80 members. If I had to pick one thing that is the most difficult, it would be this: when the conductor "leads" the orchestra he must do exactly that. He has to be a fraction of a second ahead of the players at all times, must anticipate the progress of the score, and must telegraph the beat, articulation, cues, dynamics, phrasing, inspiration, etc. as information in advance to the ensemble. It sound quite easy--until you're up on the podium trying to do all that and more effectively.