Embassies are not "part of the country the embassy belongs to, rather than the one they are in)." As the US Dept of State indicates,"diplomatic spaces remain the territory of the host state". They do, however, have special status in line with the 1961 Vienna Convention on Diplomatic Relations, and the host country may not enter an embassy without the ambassador's express permission.
While some form of diplomatic activity was carried out much earlier, the practice of states having permanent diplomatic presences abroad started to evolve in the thirteenth century between the states of Northern Italy, and expanded to other parts of Europe. For a basic overview see eDiplomat's A Brief History of Diplomacy. For something more detailed look at War and Peace in Ancient and Medieval History (De Souza, Philip, and John France, eds. New York: Cambridge University Press, 2008), or The Rise of Modern Diplomacy, 1450–1919 (Anderson, Matthew S. New York: Longman, 1993).