I would recommend reading Severed: A History of Heads Lost and Heads Found by Frances Larson, which goes into this phenomenon among others. She also gave a TED talk that discusses the issue in more modern times (specifically, the popularity of beheading videos online) but with some history for perspective.

The short version: it's very hard to answer why people do anything because human behaviour and psychology are extremely complex. However, she does discuss some factors, including how executions serve as group bonding exercises where social norms are reinforced, the thrill of the spectacle, how goverments/leaders intentionally ritualized the process as a demonstration of political control, how the executed people were dehumanized and made into icons of their (alleged) crimes (often the execution involved many steps to violate the corpse long after death had occurred), etc. She also makes the argument that it is dangerous to try to imagine people in history with our modern morality and sensibilities. People were more routinely exposed to pain and held very different attitudes towards human suffering that we do today -not that we have stopped being fascinated by death and execution or stopped watching, but that we may not do it for the same reasons.

One point that really struck home for me while reading (and then in her TED talk as well) was how people in France actually complained about the introduction of the guillotine because it was too quick and lacked the drama (and mess) of the previous executions. In fact, she argues that the move towards private 'humane' execution was at least partly in response to changes in morality and intellectuals disgusted by the 'unseemly' riotous behavior of the crowds.

At what point in time did various countries make executions non-public, and what were the exact restrictions (in the US, family members of victims can watch, for instance)?

This seems like more of a psychology question than a history question, I would recommend r/askscience