If Abraham Lincoln was anti-slavery, not racist and a republican, and Andrew Johnson was pro-slavery, racist and a democrat. When did the roles switch(since generally, now, the racism tends to be accredited to the opposite parties)?

Thanks