The hostility between Catholics and Protestants played a huge role in the execution of Mary, Queen of Scots. This answer by /u/talondearg is a good explanation of the religious upheaval of the years prior.

Mary, Queen of Scots was a devout Catholic, and Scotland was torn between Catholics and Protestants. This eventually led to protestant rebellion, Mary’s imprisonment, and her abdication on 24 July 1567, which meant that the 1 year old James was now King of Scotland. Mary escaped and made it to England in 1568, and wanted Elizabeth to help her. There was an inquiry in England to determine whether Mary was responsible for the death of her late husband, Lord Danley. Mary was neither found guilty nor acquitted, and Mary was kept in English custody for 19 years.

While all of this is taking place, Pope Pius V issued a papal bull titled ‘Regnans in Excelsis’ in 1570 that declared Elizabeth was not the rightful heir to the English throne. It gave any English Catholics permission to disobey her and threatened to excommunicate any Catholics who did obey Elizabeth’s orders. They didn’t need to pay taxes, obey the laws and so on. This inspired many plots to remove Elizabeth from the throne, and Mary was the focal point of many of these plots. The final of the plots was the Babington Plot which was a plan to execute Elizabeth and place Mary on the English throne. Babington wrote to Mary in code and told her what he planned to do, and she replied that she agreed. She was arrested, and put on trial for treason and sentenced to death in October 1586. It’s also important to note that while Mary was in England, it’s said that she was very vocal about her belief that she was the rightful heir and should be queen.

Elizabeth reportedly did not want to execute a queen as it set a bad precedent, and it could lead to Catholic rebellion. She eventually signed the warrant of execution in February and gave it to a member of the Privy Council and reportedly asked him not to do anything with it. Mary was executed on 8 February 1587. It isn’t clear if Elizabeth wanted Mary to be executed, as after she was told that Mary had been executed, Elizabeth declared it was against her authority.

When it comes to the line of succession, Elizabeth refused to name an heir throughout her reign. Elizabeth reportedly would say that the person with the most right to inherit the throne would whenever she was asked. It actually wasn’t a given that James would succeed Elizabeth. Henry VIII, Elizabeth’s father, had excluded the line of his sister Margaret (who was James’ great-grandmother) from English succession in favour of his youngest sister Mary Tudor. This was stated in his will in the event that his children all died without issue, and endorsed in the third Succession Act. It prioritized what was referred to as the Suffolk claim over any claim the Stuarts (i.e. James) would have.

Despite the other claims to the English throne and Henry VIII’s exclusion of the Stuart line, James did have a strong claim. Both of his parents were descended from Henry VII through Margaret Tudor, and James was Protestant, unlike his Catholic parents. James was a foreigner, which was not ideal, though the relationship between England and Scotland was not as hostile as it had been in the past. He was also already a King, certainly helping matters. At the time of Elizabeth’s death, James was also genealogically the closest relative. One of the other likely candidates seemed to be Anne Stanley, descended from Mary Tudor. However, the family’s religion was unclear with speculation they were Catholic, her father had been approached to be involved in a Catholic plot, and her grandmother was arrested for using witchcraft to determine if Queen Elizabeth would live much longer.

Elizabeth’s advisor Robert Cecil played an important role in James’ succession to the English throne. Between 1601 and 1603, Cecil secretly negotiated and corresponded with James, and all but confirmed that James would be the heir. Cecil requested that their communication be kept a secret from Elizabeth, that James not broach the subject of succession with her, and that James not seek parliamentary confirmation of his claim to the throne. Shortly before Elizabeth died in 1603, James was sent a draft proclamation of his accession by Cecil and James was declared the King of England within hours of Elizabeth’s death. It's not really clear whether or not Elizabeth ever specified that James was to be her heir, though she reportedly once said that only a king is fit to succeed a Queen.

Essentially, Elizabeth never really seemed to want Mary, Queen of Scots to die or that she had any real animosity towards her cousin. She wanted to protect the throne and Mary simply being in England was a threat to that. While Elizabeth kept Mary in custody for 19 years, Mary was afforded a relatively good lifestyle even while captive, including at least 16 personal staff, private chefs, and summers spent at a spa town. If Mary had not been found to be involved in one of the many plots on Elizabeth, she likely would have remained in custody. When it came to succession, James was the logical choice and rightful heir following the rules of primogeniture (and ignoring Henry VIII’s exclusion). Elizabeth had long stated that the person with the most right to inherit would, and as James was her closest living relative, it is not surprising he succeeded her.

Sources:

John Guy, "My Heart is my Own": The Life of Mary Queen of Scots.

Christopher Haigh, ‘Elizabeth I’

Vanessa Wilkie "Such Daughters and Such a Mother": The Countess of Derby and her Three Daughters, 1560-1647