Fun fact, only officers really defected to the Confederacy. I am aware of no rank and file soldier defecting after secession, and even if one or two did, there was no mass exodus out of the Regular Army.
Thats important, because Officers were held to a different standard. Firstly, many of the men who became prominent officers during the war were civilians during the secession movement. Obviously, in that weird political limbo, it was pretty easy to slip away to their state of choice. But for active officers even, there was a different code of "honor", and many of these men knew each other socially. Its not like today, where tens of thousands of officers served in active duty in the Army. With a small army, the officer corps naturally grew intimate in a way were not familiar with. And when your friend, a drinking buddy, a West Point classmate, a social acquaintance said they were heading south, it was a lot harder to arrest them for treason.
Some of them were allowed to leave at least.
Lewis Armistead for instance, resigned his commissioned in late May 1861, a month after the bombardment of Fort Sumter. His fellow officers knew full well that he was doing so to join the South. He traveled to Texas and was commissioned a colonel in the Confederate army.
I suspect Robert E. Lee is a good example of this. Trying to track down where he was when He quit the Union Army on April 20th 1861, I found that he did go to DC to accept promotion to Colonel and was courted by both sides to lead forces. When Virginia left the Union, so went Lee. I suspect many southern loyal men were in the north and made their way south.
The US Navy, courtesy of the Naval Historical Society, has a good overview of how this worked with naval officers and includes a full listing of the officers who did so and when they received an appointment to the Confederate Navy (if applicable).