During the colonial period, what would prohibit me from crossing over the proclamation line of 1763 and claiming land for the Kingdom of Bernardia or whatever?
Similarly, in the post-independence era, what would stop me from going west and doing the same?
So, first let's talk about your first question. What would stop people from crossing into the 'unclaimed' territory between the Louisiana territory and the Colonial territory? Absolutely nothing of merit. And nothing did stop them, since the territory was already being settled in an adhoc fashion prior to the Proclamation. However, this territory was still claimed by the crown, and anyone who would have settled it from the Colonies would still be subjects of that crown. Settling the area in numbers enough to defend it and claim sovereignty would have almost certainly met with the same reaction that the American Declaration of Independence did. Indeed, I don't see much difference between these two situations. The primary reason it worked at slowing expansion was that the crown had no obligation to recognize land claims in the territory. It was plain knowledge that the boundary would be extended at the direction and management of the crown, so when that happened your farm would suddenly be the crowns to dispose as they wished. Only the American revolution prevented this from being a more serious issue I'd wager.
Now, in the post-Independence era, the main obstacle to the West was the Louisiana territory for the first little while as the Indian territory protected by the 1763 Proclaimation was now de jure American territory. However, up until the Treaty of Ghent that ended the War of 1812, the Natives in this territory, at least the North, were being actively supported and armed by the British, so de facto control past the 1763 line was still spotty in many places, particularly around the Great Lakes. So you'd need an well armed and defended party to settle there. But you'd still be settling in the de jure areas of the United States. Indeed, most of North America was actually claimed even by the beginning of the 19th century, de jure at least. And the de jure part is the difficult part. So you needed to bring enough people in to both counter that de jure claim with your own de facto control, and defend against any attempt to assert the claim by the original holder.
Luckily, we don't need to guess about how this would go, we have four good examples.
The first, and most famous, is Texas. This is basically exactly what you mentioned. We have people moving in, and declaring their own country over the de jure territory of another claim. And they won! And they became their own country. And then relatively promptly, joined the Union.
Then you have California. which like Texas (and probably inspired by them) had people moving in, declaring their own country over the de jure territory of another, and they won! Well, sort of; the US invaded Mexico during it's period of de facto independence and was admitted into the Union once that war was over and the de jure territory that they de facto controlled was added to the United States.
What you'll notice here though is that, despite originally going for independence, both territories joined willingly with the United States. This is the problem with trying to set up an independent state; most people want to be part of the US, so unless you somehow set up border controls, new immigrants are likely going to agitate annexation.
Of course, there was that case where a de jure territory of the United States agitated for independence. No, I don't mean the Civil War, I mean Utah! And actually they really didn't agitate for independence, people just thought maybe they possibly might, so troops were sent in. Actually, the United States was pretty diligent in defending their de jure claims during this period, meaning sniping territory from them wasn't possible. They were also good at defending what they thought was owed to them and creating new de jure claims, but that's besides the point.
So, the biggest obstacle to saying "Screw the US, I'm making my own country!" wasn't the fact that it was already claimed by someone other than the US; the European powers were not putting much effort into defending their claims in North America by this point during the fallout from decades of war. Rather, it was Americans themselves, and not just the state. Americans wanted to be part of the United States, so even in cases where you had unclaimed territory you had Americans trying to join up with the US. This is where our final example comes in: Oregon county. An unclaimed (or rather, a territory complicated by competing and untested claims) area, it looks the way it does today mainly because it aligns with the areas of settlement by their respective eventual owners.
So, say you make a new country in 1800 in modern Kansas when it was administered by Spain but really owned by France (so really neither by neither). Great, you carved out territory from the natives with your band of men and are somehow making a go of it.
1803 rolls around, territory is purchased, and the Americans demand you join them. You say no. Maybe they have their hands full. Maybe nothing happens. 1815 rolls around and now Americans are flooding into the territory. It's not long before the original settlers are outnumbered and wham bam thankyou ma'am now you're a US State and you better like it. Or you could fight it out, like the country of Mexico did, maybe a few years later. Wham bam thank you ma'am now you're a US state. Either way, the US isn't going to just let you sit there.
Relating to your second point of post-independence, you should look into the Burr Conspiracy. Aaron Burr traveled to St. Louis after killing Hamilton and met a guy named Charles Wilkinson who he sort of conspired with to form a colony in the Southwestern area of modern day USA after defeating the Spanish. I say sort of, because he was acquitted of charges of treason by John Marshall, due to a lack of evidence. Jefferson, who was president, wanted Burr's head (figuratively), and got Wilkinson to rat him out.
Obviously, it gets a lot more complicated than that. Burr's motives and actions prior to his 'conspiracy' were extremely interesting. But, essentially, Jefferson feared a secondary colony that would cause problems for a rapidly expanding America.
I would recommend Nancy Isenberg's biography on Aaron Burr and Henry W. Brand's The Heartbreak of Aaron Burr for further reading. This whole situation was a fantastic example of why it was damn near impossible to simply "start your own country"