Part of your confusion streams from a common misconception the Federalist proto-party that emerged around 1793 in support of Hamilton's policies was not the vague faction that had fought for ratification of the constitution in 1787-1788. The groups that fought for or against ratification were formed almost entirely on a state or local level and comprised a wide range of beliefs regarding the constitution they had no national legislative agenda or set of beliefs aside from ratification and even that often had differing opinions. There were many moderate-antifederalists who favored a stronger federal government but had concerns regarding the constitution in its current form (often concerns about unlimited terms for the President) you can put Monroe and Jefferson into this group, eventually many of this group will support ratification but with amendments. Then you had a smaller group of anti-federalists who wanted the states to cede little to no power or had other concerns and wanted a completely new convention, you can put Patrick Henry into this group. Finally you have those who favor the constitution but not with amendments Hamilton fits here. Of course this is ignoring that many people voted on ratification entirely for local reasons or to support the local elite. After ratification the anti-federalists tried to win the upcoming elections but they were dominated largely by federalist candidates. Once the threat of the Anti-Federalists had passed by 1789 the federalist faction was largely played out.

Fast forward to 1793 differing visions of America's future had caused two proto-parties to emerge centered around Jefferson/Madison and Hamilton/Washington. Both sides attempted to claim the term "Federalist" so they could paint themselves as the true defenders of the Constitution, when it didn't work out for Jefferson they adopted "Republican" as the defenders of Republicanism. It sounds nice and simple but the reality was in the elections of 1794 both proto-parties had 30+ names for them in the local elections. In Delaware for instance there were Federalist-Republicans running and Republican-Federalists!! Even after the election of 1800 Hamilton wrote his colleagues saying they should try to win the term Republican back.

In regards to the anti-federalists and federalists of ratification days 1787-1789, they went to both parties although Jefferson noted that more tended to gravitate to his party. Ultimately both Federalists and Republicans were largely made up of those who had supported ratification. As to why Madison didn't side with Hamilton and later backed the Virginia resolution, there are probably close to half a dozen prominent theories that would take some time to dive into. I would recommend reading a short essay by Gordon Wood entitled Is there a James Madison problem?

Hope this helped!