The ordinary infantry would have remained with their lords, the barons, in England, when they stayed to reap the rewards of allotted lands that the new king, William, doled out to them.
Only when they, or the king, returned to France or Normandy would they have gone with them.