Yes, you are right, of course ! What is difficult for one person may not be for another and what is easy for one person may not be for another. This is why the level of difficulty placed on pieces is only relative and is meant to give just a general idea on when/how it should be tackled. Of course, there are particular organized systems that have piano students passing from "level" to "level" and within each level, there is accepted repertoire. I think that type of system can be some kind of light "guidance" or so, but if taken as the be all and end all of pianistic advancement, it's also a really good way for a pianist to develop "gaps" and "holes" in their musical understanding and proficiency.
No matter what, each piece will present its own particular type of language or use of the musical language. Often the RH and LH have a kind of individual dialect or so, and a teacher or an aware student will be able to pick out what will be the particular difficulties of the piece. Ideally, particular challenges would be addressed in a way that the challenge can be dissected and focused on bit by bit.
These difficulties may be addressed in preparation pieces, too (and if exercises are being assigned, it should be with specific goals of preparation in mind and hopefully it will relate to a particular piece that is being learned at the time). Learning an invention (or THE inventions), for example, is a good preparation for fugues since perhaps the biggest challenge of counterpoint is in thinking in the form of more than one complete voice at a time (and hands have to coordinate in a particular way with counterpoint), aside from whatever technical challenges each "voice" may individually present in particular areas of the piece.