Rules of succession vary by culture and/or the type of law used by the kingdom in question. If we're talking strict primogeniture, which is probably the most common and best known type of inheritance, then succession would pass down the eldest son's line before passing on to the next child.

One of the most famous examples of this actually happening comes from the succession after the death of Edward III. Edward III's eldest son, Edward the Black Prince, died before King Edward did but not before having a son. That son became King Richard II instead of any of the Black Prince's brothers.

Game of Thrones is a work of fiction, though, so I wouldn't expect it to necessarily follow reality. Sometimes in the Middle Ages individuals would dispute what type of law applied to a disputed succession. The most famous of these is probably the dispute between John Balliol and Robert Bruce the Elder (among others) about who should inherit after the Scottish King Alexander III died without any clear heir (his children having all died already). John Balliol had a better claim by strict primogeniture but Robert Bruce claimed that by Roman Law he was closer to Alexander III since he had fewer degrees of separation to the deceased King (Bruce was a generation older than Balliol).