Terminology note: a crest is an object to put on top of your helmet, not a two-dimensional image to put on your surcoat. You mean "coat of arms" or "arms" (those are synonyms), and personally I'd let "heraldry" slide in a casual conversation.

The coat of arms that you would put on your shield, your horse's barding (cloth), et cetera, would be your own coat of arms; you had no more than one, and even that early, I suspect that you would have one (or were required to have one before you could enter the tourney). You'd absolutely not bear your lord's arms.

It's more complicated to say what a person's coat of arms were. It's sometimes said that a coat of arms said "this is me" and were unique to one living person. That needs to be qualified. A wife could bear her husband's arms (though perhaps with her father's arms included). I believe unmarried daughters would, or all daughters if daughters could transmit arms and the man had no sons.

A herald representing the lord would wear the lord's arms: that's the only case I know of where it was unconditionally proper for man Y to bear the arms of man X.

So the proper thing was for younger brothers to have their own arms, which might or might not be the arms of the father with one or more changes ("differencing", "cadency"). But like most property rights, much depended on the owner of a right taking action to prevent infringement, so if a man saw his younger brothers (or someone unrelated, for that matter) bearing their father's undifferenced arms, and if the man didn't take them to court, in practice the others would get away with it.

If you inherited several coats of arms -- for example, your mother was the heiress of a lord, or your father was the heir of someone like that -- then you could divide your shield into rectangular panels, each displaying one inherited coat of arms ("quartering"). If your wife is an heiress, you might put her arms on a small shield on your coat of arms ("arms of pretense"). I don't know at the moment whether, in France, they had arms of office for other than bishops and major abbots, and if so, whether you'd place the arms of office on half of your arms ("impalement").

But to go back to the answer: you had your own coat of arms and you'd display that.