1/2
A series of very incremental reforms based on the militia not being a great tool of the state is at the root of the change. Sharp observers will not that "not being a tool of the state" is pretty much the whole idea of a militia in the first place, and roping the national guard units to the same chain of command as the sanding US forces is... not a militia.
But we should also note that the ur-political theory of militia and the legal framework of the militia used by the United States is emphatically not the same thing. However, in the early stages of US development as a state, use and reform of the militia was a hotly debated topic. I won't give a long and exhaustive list of the changes in the militia over the course of the 19th century, but I'll give some of the earliest legal definitions and a few examples of their use.
Militia in North America before the Constitution
A militia is a social institution, first and foremost. It is not a military institution; or rather, it is a military institution by necessity but not necessarily in intent. Militias reflect and reinforce the social structure of their local communities, made up as they are of citizens (remember that this means property owning men) with an interest in their local communities: a stake. The property requirement was seen as a reflection of a man's shared responsibility to safeguard the community, since safeguarding the community also means safeguarding his property. It is also, as you can easily see, a way to restrict membership to a particular class.
The militia has duties and responsibilities that relate to safeguarding the community. They are the primary firefighting force, they are the primary peacekeeping force, they are the primary law enforcement force used in conjunction with sheriffs, marshals, or other elected or appointed law enforcement officials. In law enforcement you also see a reflection of the community. In the south, the militia was embodied as a "slave patrol" to catch escaped slaves and to discourage slave uprisings. Obviously, that wasn't done in non-slave states, though repressing rioters and defending property was always an element of their organization.
However, the militia is not as clean an organization as all that, it is at its core a reflection of its community, and sometimes that community organizes and embodies as a means of active social resistance. Though written into the United States legal doctrine as a force to repress insurrection and quell riots, when the "respectable" (read: propertied) members of that community decide to protest, they do it as a militia. Militias were a large element in what we call "politics out of doors," and most of the organized political resistance of the British government during the lead-up to the War for Independence was done as militias, or at least with militias in support.
During Shays's Rebellion, the militia was so well organized they kept pay records and muster reports, wore uniforms, and addressed their leaders as officers. This is probably no surprise, because most of them were veterans of the War for Independence and, all of them being propertied men were not the rabble of poor men as is often represented. The force that was mustered to confront them was also militia: the only difference between the forces was that one was lawfully embodied and the other wasn't.
Militia in US Law
Shays's rebellion caused a moment of rhetorical panic among the American framers; while most of them would have at least acknowledged the underlying need of having a militia instead of the monstrous machine of a standing army, they also perceived a problem with leaving the arm of the state in the arms of the citizens: what if they embody to resist laws? To be fair to the framers, some of them saw this as a positive thing. It was just another means of asserting political will. But the 18th century conception of good governance prized order above most other concerns, as order was the antidote to chaos and anarchy. Letting farmers or rope-makers determine what laws were just and which were not was anarchic and dangerous, and so firm guidelines about the legitimate embodiment of the militia were debated and written into the constitution.
The constitution addresses the militia in Article I, Section 8:
To provide for calling forth the Militia to execute the Laws of the Union, suppress Insurrections and repel Invasions;
To provide for organizing, arming, and disciplining, the Militia, and for governing such Part of them as may be employed in the Service of the United States, reserving to the States respectively, the Appointment of the Officers, and the Authority of training the Militia according to the discipline prescribed by Congress
Militias were subject not to federal authority here, but to the states: congress had the ability to call the militia in times of crisis (to enforce laws, repel invasions and suppress insurrections), but the day-to-day operations would be carried out by the states.
In Article II, Section 1:
The President shall be Commander in Chief of the Army and Navy of the United States, and of the Militia of the several States, when called into the actual Service of the United States; he may require the Opinion, in writing, of the principal Officer in each of the executive Departments, upon any Subject relating to the Duties of their respective Offices, and he shall have Power to Grant Reprieves and Pardons for Offences against the United States, except in Cases of Impeachment.
The president is the general commander of the militia when it's embodied as several states, such as when it was embodied during the Whiskey Rebellion, for instance. I wrote in much more detail of that conflict here.
And then there's the Second Amendment, which I'll quote in full here:
A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed.
Briefly, the idea of "well-regulated" meant many different things to different framers: some meant it as the understanding tha the militia will be subject to a rigorous command and control mechanism that ultimately reports to the preisdent, others that it meant that states should have to have their leadership scheme subject to approval by congress, some that it was organized and disciplined along a congressionally approved trajectory, and some just that it regularly drills.
But there are more laws, in the Militia Acts: The First established the legal framework in which the president was authorized to embody the militia, and the Second set down the framework in which this militia was to be organized.
Now again, we should discuss the tension between the militia theory and the militia laws, because they were different. The laws here are pretty clear that the militia is ultimately subservient to federal authority, and any organization of men without sanction by either their state or the federal government are not legal embodied militia, they're rebels. But then, even as rebels, they're organized and behave like militia, as Shays's Rebellion and even the Whiskey Rebellion proved.
During the War of 1812, the militia proved so keen to defend their ideal embodiment that they regularly refused calls for invasion, citing a still-debated question about whether militia could be compelled to serve "overseas" or in foreign territory. Officers in the various invasion armies circumvented the question, ultimately, by asking for volunteers from the militia instead of forcing a confrontation by ordering them to do so. I want to emphasize this, because I think it's important: general officers had to ask their men to accompany them on invasions, because the reality was compulsion would lead to a confrontation they were not guaranteed to win.