In 1820 the Missouri compromise was agreed that stipulated states above the 36.5 parallel excluding Missouri would be free states while those below this line would be slave trading states.
Given this why was California entered into the Union unpartitioned and free in seeming breach of the 1820 agreement? Also why were New Mexico and Arizona allowed a vote on the matter?
The shortest version of this is that the Missouri Compromise doesn't apply. This is the last section of Missouri's statehood bill:
SEC. 8. And be it further enacted. That in all that territory ceded by France to the United States, under the name of Louisiana, which lies north of thirty-six degrees and thirty minutes north latitude, not included within the limits of the state, contemplated by this act, slavery and involuntary servitude, otherwise than in the punishment of crimes, whereof the parties shall have been duly convicted, shall be, and is hereby, forever prohibited: Provided always, That any person escaping into the same, from whom labour or service is lawfully claimed, in any state or territory of the United States, such fugitive may be lawfully reclaimed and conveyed to the person claiming his or her labour or service as aforesaid.
The bolded portion if the famous slavery ban, which Americans all hear about in school. It's not a sea-to-sea line of any kind, but only one that applies specifically to land acquired through the Louisiana Purchase. It excludes the Texas, the Oregon Country, and the Mexican Cession. So there's no rule in place that clearly covers the land come the time of the Mexican War.
That's the rub, in fact. Mexican law abolished slavery in the 1820s, at which point Mexico included Texas (even the Texans thought so) and the Mexican Cession. In the normal order of things, there would be a presumption that that law continued until changed by some US authority. As soon as James K. Polk wins one for slavery by annexing Texas and sparking a war with Mexico, which he worked to engineer, antislavery politicians get together with the idea that they'll do something to affirmatively preserve freedom in any lands taken.
This isn't a place where slavery already exists, and all the new land (except Texas) will come into American control as territories. Congress has sweeping power to do anything it likes in the territories, including ban slavery. Exercises of that power are actually older than Congress itself, dating to the Northwest Ordinance. Everybody in power, up to and including John C. Calhoun, has lived with that power in theory (if not always practice) for their whole lives. As a member of Monroe's Cabinet, Calhoun even signed on to the Missouri Compromise's constitutionality. Since there's no existing slavery (except in Texas, which they're calling a mulligan on) and no prior settlement relating to slavery except for a law of Mexico excluding it, doing a slavery ban seems like a relatively achievable thing.
David Wilmot (D-PA) gets up in the House and proposer a rider on an appropriations bill. The bill is to fund negotiations and a peace settlement with Mexico, so his rider is fairly germane. It doesn't say anything about what Polk can or can't negotiate for, except that in any territory taken from Mexico in the war (so not Texas, which per the US became American territory through a joint resolution annexing it prior to the war) but does say that in any land taken slavery must be excluded. This is the famous Wilmot Proviso. It uses language identical to the Northwest Ordinance and repeated in the Missouri Compromise, completely orthodox stuff.
Cue a multi-year storm of controversy including threats of secession, pistols drawn (Thomas Hart Benton (D-MO) started walking up to Henry Foote (W-MS) in the Senate. Benton was a big, aggressive dude. Foot drew on him.) and harrowing rhetoric. The sections are clearly irreconcilable and something has to give. California's freedom is part of the compromise to pull back from the brink.
There are a lot of moving parts here, but I'll try to stick to your questions. For all the Mexican Cession, the Missouri Compromise doesn't matter. Extending the line was put forward as a conservative remedy to the crisis, but rejected by both sections. So was a revision that would move the line a bit south for California. That territory, thanks to the gold rush, had a rapidly growing white population that greatly exceeded the ability of the provisional military government to manage. It needed its own governance quickly. The Californians, without the knowledge of Washington, got together and wrote themselves a constitution, asked statehood, and specified the modern boundaries of the state. They chose overwhelmingly to be a free state.
So what are they going to do? Force slavery on California? Such an attempt would probably be impossible to justify to the northern populace (it's about as outrageous as you can get for them) and violate longstanding beliefs about the sanctity of self-government and popular sovereignty (for white men) which are especially sacrosanct to the ruling party when all this starts, the Democrats. It's a non-starter, though there are serious proposals to divide California and get a slave state in the south half of the state which go on into 1860. Even if you could manage politically, how would one set up a slave system in CA when it's an arduous trip across desert and mountains to get there? (I know of one plan to try, but it's not plausible. If tried it probably would have killed a lot of people.)
In the end, CA's statehood without slavery is taken as a kind of fait accompli. The white South hates it with a fiery passion, but there's not a lot to be done in the near term. But if they've got to suffer that, then they want some kind of compensation. The Fugitive Slave Act is part of it. So is buying the votes of Texas' senators by paying off the Texas Republic's debts by means of buying its not that credible territorial claims back to its modern boundaries. Those claims went out to the Rio Grande...in the west. That means half of modern New Mexico and then some, vast tracts of land that the Texan government had never exercised any meaningful control over.
Paring back Texas, if somewhat less than originally proposed, creates a legally clear space between it and California. The main white community in that area is the LDS church in Utah, which is on poor terms with the US government. That land is organized as two territories, Utah and New Mexico (Arizona is later cut out of New Mexico territory, Nevada out of Utah territory) and here's where we get a real mess.
Textbooks will tell you that the decision was the people could vote slavery in or out there, a brand new policy innovation called popular sovereignty. (Which is a term used for a lot of related concepts, unhelpfully enough.) The problem is that's just not so. Whether or not the territorial governments could decide for themselves on slavery, and if so when, is the subject of controversy from the get-go. When asked directly if the draft bills for them instituted a right to do so, the architect of both, Stephen Douglas, told the Senate that his own Committee on the Territories had no consensus on the question. The ambiguity was part of what made the solution appealing. Northerners voting for the territorial bills could say they hadn't voted to extend slavery. Southerners could say that they had. There's probably an assumption that slavery will eventually go into New Mexico but not Utah, since splitting up jurisdictions tended that way, but that's a matter of how things fell out in the end more than a firm policy.
It's dubious to say that a policy is enacted when the people enacting it are themselves deliberately unsure, but we can bring this back around and vindicate the textbooks and survey courses a little bit. Later in the 1850s, both New Mexico and Utah territories vote slavery in. In doing so, they demonstrated that popular sovereignty (if it applied at all) meant slavery got in and freedom was kept out. That fit with the whole sweep of the antebellum, where absent specific, adverse legislation slavery went everywhere. It also went to many places we don't usually view as having slavery, because restrictions on slavery tended not to be enforced or easily sidestepped...but that's getting quite far afield.
Sources
William W. Freehling's Road to Disunion, Volume 1
James McPherson's Battle Cry of Freedom
Michael Morrison's Slavery and the American West
William Parrish's David Rice Atchison of Missouri: Border Politician (This is obscure and only important if you want to read more about the Foote-Benton confrontation.)
David Potter's The Impending Crisis
Elizabeth Varon's Disunion!