Do we know how or why marriage ended up the near-universal way of managing human social relations across major civilizations? Did marriage ever have major competition from a different arrangement?

This is really mostly an anthropology question, but there may be some value in looking at how marriage manifested in ancient civilisations around the world. I think I can answer the first part of your question with some comments on prehistory and cultural anthropology. After that, I will provide some examples to illustrate varying norms within major world civilisations and offer a possible answer for the second part of your question.

Marriage is considered a 'cultural universal', but while we "know it when we see it", marriage doesn't always take the same form. Fortunato (2015) provides a useful definition that applies broadly:

a socially recognized union, normatively endorsed, between two or more individuals. This may involve (i) restrictions on sexual relations, typically allowed exclusively between spouse(s), and (ii) the investment of resources in any children associated with the union. Resources may be “social” (e.g. political status) or physical (e.g. material property), and they may be invested by the spouses themselves and/or by their kin. Where the resources involved are substantial, marriage may be linked to the notion of legitimacy, which serves to channel them to “appropriate” individuals.

As you can see, this leaves us a lot of room for variation. For example, all societies contain some proportion of monogamous marriages, but the majority (93% in the HRAF, with males in 70% preferring polygyny where possible) also permit polygynous marriages (one man and multiple women) and a small number (7 out of 1,167 in Murdock's Ethnographic Atlas) permit polyandrous marriage (one women and multiple men-- in all these cases, these men are brothers). [Note: It can be misleading to label a culture as 'monogamous' or 'polygamous'. Only about 5-10% of men in cultures that permit polygyny have more than one wife.] Norms around extramarital sex also vary and there are often different standards in this matter depending on gender.

It's difficult to reconstruct prehistoric marriage systems from archaeological and genetic evidence, but in their very useful study on male and female gene flow at the Neolithic transition, Rasteiro & Chikhi (2013) suggest using data from burials and ancient DNA that this period saw a major reduction in the incidence of polygyny and increased frequency of nuclear, monogamous families. When looking even further back the situation becomes even more difficult to resolve, but based on cross-cultural data we presume that monogamy and polygyny existed in the early human environment.

The answer to the first part of your question is, then, not too complicated: due to a common origin. All these civilisations emerged among groups that already had marriage, though its nature and how it connected to the society as a whole varied over time and space.

In Early Dynastic Sumer we find no convincing evidence for polygamy, though there is a questionable reference that I have previously discussed here. Marriage was a contract made by the agreement of the heads of two families; the family of the groom would bring a gift of 'bridewealth' to the head of the bride's family after the contract was agreed. The taking of the bride's virginity was an important part of the ceremony and the text of the Stele of Ušumgal suggests her virginity was confirmed by multiple witnesses. The bride also had a dowry, given to her by her family, which she could sell or bequeath as she wished. She could draw up contracts independently or jointly with her husband, and she could hold property and land in her own right separate from those of her husband. If a man died his widow inherited the administration of his estate until his sons came of age. Here, marriage is the framework within which sexual (implicitly reproductive) and inheritance-related human activity takes place and it has major social and economic import.

In the Ur III period, we find evidence of a king (Shulgi) with a queen and 5-10 lukur, or concubines. Separate quarters for women which contained the queen, the concubines, and many other female staff are well-attested from this point on, with examples in Ebla, in Old Babylonian Mari, and the Neo-Assyrian and Neo-Babylonian Empires, as well as among the Achaemenid Persians. The queen had an important role in the state that could not be divided up between many women, but there could also be several other junior wives-- 7 or 8 in the case of Zimri-Lim of Mari. Sending junior wives to the ruler enabled the great families of the realm to link themselves to the king, and it could also fulfil a diplomatic function.

There is an unfortunate lack of evidence about marriage in Old Kingdom Egypt, but the situation improves in later times. There are some possible examples of polygyny in the Old and Middle Kingdoms, but only among very wealthy individuals. Adultery was punished by removal from office, exile, or execution. In the New Kingdom, the family appears to have been considered a juristic entity in its own right. A man's offspring by his wife were his legal heirs but she did not come under his legal control, and both partners were expected to fulfil their mutual obligations. Daughters received their inheritance as a dowry when they married but sons only acquired their inheritance on the death of their parents. Monogamy was the rule though powerful men might have many wives, concubines, and children; Ramesses II is supposed to have had about 50 each of sons and daughters.

Among the Hittites, a Bronze Age Indo-European people who ruled much of Anatolia and Syria, marriage was always monogamous. It was formed in three stages: the parents of the bride (both of them) would promise to give their future son-in-law their daughter and would in turn receive a gift, then the parents 'bind' their daughter to the man in exchange for a further betrothal payment. The parents and the man can both withdraw until the marriage has been consummated, by breach of contract by the parents incurs a repayment of three times the betrothal payment and breach by the groom means he loses the payment he made. Finally, there is the wedding and consummation which results in a legally complete marriage, only soluble by death or divorce. Part of the ceremony is the transfer of the dowry to the bride by her father.

We're seeing a lot of common themes: marriage acts as the main context of sex and reproduction, it defines social and economic roles for the marriage partners, and it provides a framework for inheritance and the relationship the participants have with their parents.