I would point to the Treaty of Westphalia and the concept of Westphalian sovereignty as a key moment in the concept of a nation. This was really where the idea of a nation state as a formal abstraction took hold. Key to this is the idea that no external power had any role inside of the politics and society of a nation. It also codified the idea of territorial integrity, which has served as the lynchpin for most major conflicts after it.

The next two major steps where the rise the concept of civic nationalism - most notably with the United States where a group of people embracing the same concept of civics qualified to be a nation, and then that of ethnic nationalism.

The reformation helped lead to the collapse of the last vestiges of the holy roman empire, and led to a weakening of the central authorities that had been present before - the church in rome and the empire. Into this void stepped absolute monarchs and absolute states, which took power that previously had been in the church, and assigned them to the crown or the nation. This, combined with the concept of ethnic or civic nationlism led to states that had identities beyond the historical role.

I agree with /u/inthearena that Westphalia was a big milestone in state formation.

Charles tilly is a good place to look. He tries to explain why the nation state became the predominant form of political organization in Europe. He argues that polities were formed primarily for protection. As wars became more expensive and armies more complicated during the middle ages, other polities became obsolete. For example, city states were too small and weak to defend themselves while larger empires were too disorganized. Thus, the modern nation state came to be. His argument is usually summed up: "War made the state and the state made war."