The definitive answer didn't come until 1869, with Chief Justice Chase delivering the opinion in Texas v. White that secession had not been a constitutionally valid act, citing the perpetual union clause of the Articles of the Confederation and the language of the Preamble to the Constitution as clear enough evidence of what the latter document intended. Specifically, the argument was essentially that "in order to form a more perfect Union" carried over the perpetual Union clause, as any attempt to perfect a union would at the very least transfer the perpetual nature of the preceding union. This doesn't address your question, I'm just adding that for anyone unfamiliar.

To give a brief summary, the positions for secession were that the federal government consisted of a compact between states, those states retaining sovereignty and thus a right to sever its ties to the compact of states it entered into. That sovereignty retained, it was argued, by Calhoun among others during the Nullification Crisis, that the power to also nullify federal policy, with secession a valid reply to any extreme case of federal enforcement of an unjust policy. The most relevant, direct debate that touches on this issue are probably the Webster and Hayne speeches, which are also convenient because we have Madison directly weighing in on them via a letter to Daniel Webster in 1833. In it, Madison distinguishes the right to unilateral or "at will" secession from the right to separation through revolution under "intolerable oppression," though interestingly uses the term 'secede' to describe either case. Nevertheless, he very quickly and directly adds:

The former [right to unilateral secession] answers itself, being a violation, without cause, of a faith solemnly pledged.

He affirms in the subsequent sentence that the latter issue is not one of any "theoretic controversy," implying that it is retained by any people as a natural right. He then expounds on the debate over how the Constitution was formed, whether as a constitution of the sovereign people directly, or entered into by the people via a compact between sovereign states. Madison affirms that it was made "by the people...imbodied into the several states" and "therefore made by the States in their highest authoritative capacity." He also carefully notes that in every operational sense either side of the present debate is indistinguishable, for the document was created by an "authority...equally Competent in both." He does draw one distinction that only becomes relevant if the union is dissolved. That difference, he explains, is that any dissolution of a league or confederacy formed by states would leave the people in a state no different from when the compact was entered into. If the other case, "if formed by the people as one community," dissolution would "reduce them to a state of nature, so as many individual persons."

What much more clearly is argued against in that letter is nullification, as Madison emphasizes that the way in which the Union was formed has no bearing on whether policy issued by the federal legislature, which he affirms to be supreme (as done in explicitly in Article VI of the Constitution), can be declared void by an act of a state's legislature so long as the union remains undissolved. This might suggest, then, especially with how Madison concludes the letter, than any dissolution of the union would be better for the people if done through the states. However, I don't see that as a justification for the unilateral secession of a state, particularly given how Madison refers to at-will secession earlier in the letter. Further, the fact that he contrasts it with separation in the case of intolerable oppression seems to suggest that, if anything like a secession movement is to be attempted, it should not be all that distinguishable, causally and functionally, from invoking a natural right to separate over a unconstitutional and unjust law.

As for the perpetuity of the Union, relying yet again on Madison, it seems that many at the time (at the very least) understood that ratification should and would be treated as a permanent entrance into a union. In a 1788 letter to Hamilton, in reply to questions Hamilton encountered during New York's ratification convention, stated rather unambiguously:

The Constitution requires an adoption in toto, and for ever. It has been so adopted by the other States. An adoption for a limited time would be as defective as an adoption of some of the articles only. In short any condition whatever must viciate [sic] the ratification.

So it's very clear that Madison believed that the states were entering into the Union, at minimum, under the assumption that they could not back out. I'm hesitant, however, to use this as a concrete answer to first part of your question, as Madison then continues:

What the New Congress by virtue of the power to admit new States, may be able & disposed to do in such case, I do not enquire as I suppose that is not the material point at present.

This might actually shed some light on your follow-up questions, as to me it seems to suggest that this issue could be decided in the future when the concern actually arises.

Of course, Madison's opinion alone would not be a definitive answer automatically, but more than the other founders it seems relevant to include given his pronounced role during the drafting of the Constitution, his authorship of the Virginia Resolution, presidency during the 1812 War, and his having lived to observe and comment on the Nullification Crisis.

I'm sure /u/Irishfafnir or anyone else with a greater specialty in this period will have more to add, but hopefully this is a good start.

At the framing of the constitution the issue of secession was very much intentionally left out of the debate. If secession was explicitly allowed, then the federal government would be too weak since any state could just secede at will. If secession was explicitly not allowed, then states would be hesitant to enter into a permanently union in the first place.

The issue, had it being brought up, had the potential to simply make an agreement over what sort of constitution the nation should have a non-starter. Therefore the framers at the constitutional convention intentionally left the issue ambiguous.

Would the tenth amendment give them the right? Nowhere in the constitution does it say that states can't leave, just how they can enter. Wouldn't that mean the states have the power to decide if they can leave or not?