My understanding is that Lincoln warned states in rebellion that he would emancipate slaves under their jurisdiction if they did not rejoin the Union. But the whole reason those states were in rebellion was because they feared the Federal government would pursue abolition. So it sounds like from the point of view of a rebel state, the options were:
1) Rejoin the union and see slaves freed by abolition. 2) Stay in rebellion and maybe see slaves freed by emancipation.
Those two options kind of look the same.
Did Lincoln actually believe this would be an effective way to motivate rebel states to rejoin the Union? If so, why?
Motivate states? No, Lincoln doesn't mean that. He has in mind loyalist whites and wavering whites within the rebelling areas as well as the legal basis on which he needs to operate. He lays it out here, in the final document:
Now, therefore I, Abraham Lincoln, President of the United States, by virtue of the power in me vested as Commander-in-Chief, of the Army and Navy of the United States in time of actual armed rebellion against the authority and government of the United States, and as a fit and necessary war measure for suppressing said rebellion
Keep in mind as we go that Lincoln does this on his personal authority as C-in-C and during a time of armed rebellion, as specifically a military necessity. That's what he's thinking from the start and it shapes how everything shakes out.
So let's rewind back to summer:
The Emancipation Proclamation, or something like it, was first floated by Lincoln to his Cabinet in a meeting on July 22, 1862. This was his second round with them discussing similar war measures, which had previously run on the lines involved in the Confiscation Acts. Those laws permitted the seizure of property from those deemed disloyal. There was a judicial process for ordinary goods, but the seizure of enslaved people was self-enacting. Lincoln was uneasy with that at the time, signing the law but also sending to Congress a draft veto message for it. The day prior, he ended with an order to facilitate colonization of freedpeople abroad, which the Cabinet was cold on.
The meeting on the 22nd has Lincoln open with a new order, all of three sentences. The first one a warning: sixty days to quit all that treason or face general forfeiture of property under the Second Confiscation Act. Essentially, Lincoln's telling them that he means to enforce that law. Number two: Lincoln still supports emancipation on gradual, compensated terms. In other words, buying the enslaved their freedom over a long time.
The one we want is #3, which is in the same vein with the others but goes waaaaaay farther: Citing his authority as commander-in-chief, under his war powers and thus independent of the authority of Congress, the Confederates had until the first of the year or all their slaves went free. That's immediate, uncompensated emancipation that would roll into practical effect as soon as a US army took control of an area. Nor does it make a distinction between loyal enslavers, who get rewarded with a right to keep on enslaving, and disloyal ones who are punished by forfeiture of the lives they own. Loyal and disloyal alike would feel its bite, and keep on feeling it as the war progressed.
The Cabinet divided on the proposal. Some said nothing. William Seward thought the war itself was already destroying slavery, so why bother? Others feared that it would produce a general revolution in the slave states or cost the GOP dearly in the upcoming elections. Word leaked out, but Lincoln didn't immediately blaze forward. Instead he waited on what he could call a military victory, at Antietam, so it wouldn't look like a desperation move. That's when he announces, complete with the deadline.
Does he expect that deadline to convince state to flip? No, but because Lincoln is doing this as an exercise of his war powers -and one he's a little hazy on the constitutionality of himself- he needs to clearly connect it to the war. He attacks slavery not on the grounds that he hates it and wants it gone, though he does, but because it provides material support to the rebel war effort. On those grounds, Lincoln thinks courts might uphold the action. He's sure they will not if he just declares it out of concern for human rights or other benevolence. If he just makes a general proclamation without something like that as both advance notice and contingency for his pressing on, it looks more like something he'd just have done anyway.
Likewise Lincoln exempts areas where he believes he might flip disloyal whites or bolster loyal whites, independent of the foregone conclusion that he's never going to change enough minds to shift a state's existing rebel government. These are chiefly places where the US has maintained military control for some time and where loyal provisional governments are in operation. That's why Tennessee, though large portions of it remain in traitorous hands, gets a general exemption. He's got Andrew Johnson, the most disastrous president in US history, running the state and Johnson needs some cooperation from enslavers to do his job. Conversely, Lincoln doesn't bother exempting areas with few white loyalists, but sound US control like the Sea Islands. They have instant emancipation because there's no white loyalty to win. His decision not to exempt eastern North Carolina prompted the resignation of its military governor, who took that as a commentary on his efforts to restore loyal government.
Furthermore, the military necessity justifies Lincoln's departure from his previous commitment to gradualism and compensation. The war needs winning now, not in a hundred years. The US also needs more soldiers, so instead of rejecting black men bearing arms entirely he invites them in. That gives Lincoln a twofer: He's taking enslaved labor away from the enemy and putting it to work to defeat the enemy, directly. That entails a radically different vision of black Americans' future than he had previously considered and which he declined to explore. Lincoln obviously had thoughts about it, and would continue to do so, but they're beyond the scope of what he tailored as a precise wartime measure. That kind of unanswered question, and continued doubts about whether emancipation could endure after the war ended, sets Lincoln on finding a more durable solution to the slavery question in the form of a constitutional amendment.
Where are you getting this idea that Lincoln threatened emancipation to try and coax the seceded states back into the Union? The Emancipation Proclamation was a war measure, and done strategically and lawfully. Lincoln (as an attorney) understood the law, and so understood his Constitutional ability regarding seizure of property. By considering slaves in Confederate hands as "contraband," he could justify legally freeing them, in order to be able to use their manpower for the Union. The latter part of the document is an offer for enlistment, which some 180,000 blacks eventually did. The Proclamation also deliberately excluded areas not considered to be in rebellion at the time, to include KY, TN, MO, DE, MD, and parts of Louisiana and Virginia. The areas not mentioned maintained slavery until passage of the 13th Amendment in 1865. So slavery still very much existed in places like Kentucky after Emancipation, because they were not officially part of the Confederacy. Lincoln recognized that for Southern states to rejoin, it was be done via military action. He would not budge on the slavery issue, especially after 1863, and if the Confederate States were willing to leave the Union before he even tried to do anything regarding slavery, they certainly weren't about to believe him if he offered them some sort of slave concession to rejoin. By early 1865, the only real stipulation to surrender and to begin Reconstruction for Southern states was through passing the 13th Amendment. So far from Lincoln trying to coax them back in, by early 1865 it was more of, "you ARE coming back to the Union, and you'll do it without slavery. If you concede to this, we can proceed, if not, the war (and by this point total war) will continue."