The desert fathers emerge toward the end of the fourth century as Christianity transitioned from a fringe cult to an accepted community in the pantheon of faiths. While earlier Christian communities had faced persecution and martyrdom, in this new period asceticism became the hallmark of true commitment to the faith. Thus some Christians sought to lead a life marked by exceptional ascetic rigor while pursuing constant prayer and fighting the temptation to return to the world. These figures spent their time alone in caves, atop pillars, or in the ruins of old roman structures now reclaimed by the wilderness and often gained a reputation for sanctity and holiness in their local community, becoming—ironically—something of an attraction and public figure. St. Anthony, whom historians consider to be the first of these "Desert fathers," retreated to the desert sometime in the late third century after giving away a large inheritance he had recieved from his parents. As a hermit living in an old military outpost, his commitment to prayer and the memorization of the bible attracted a number of followers that he organized into a informal monastic community that continued until his death in probably 356.

Some later emulators took Anthony's message of retreating from the world a little too seriously. My favorite example is Saint Symeon the Stylite, who, frustrated at the incessant visitors who came to him for prayers, placed himself atop a pillar to escape their pleas.

The communities that formed around these Desert Fathers became the earliest monasteries, ordering themselves around a communal set of principles. These "Rules" by which they lived their life formed the backbone of the monastic movement as it spread across the Late Antique world, although the earliest focused on physical work and communal prayer, not the intellectual work and reading that later monasticism would become known for.

This is merely a cursory sketch of the very large question you have outlined. I might suggest some further reading if you feel interested. Peter Brown's The Rise and Function of the Holy Man in Late Antiquity (1971) remains the fundamental text for understanding the growth and social importance of these figures. For a more general overview, I would still recommend Peter Brown. His World of Late Antiquity is now in its Third Edition, and has an excellent chapter on the desert fathers and the growth of monasticism.