The short answer here is that New Jersey's legislature had a significant number of Copperheads/Peace Democrats among its number, and they tended to bitterly oppose Reconstruction efforts. The new legislature elected in 1864 was not seated until the Fall of 1865, so it was a mostly Copperhead legislature that voted down the Amendment, and a more solidly mixed legislature that then ratified it (I do not have full numbers of each party delegation after the 1864 elections, so perhaps another Redditor can delve into this more deeply).

With slightly more background: New Jersey, through a combination of mutually-beneficial businesses and cultural idiosyncrasies, had a somewhat more pro-Southern outlook than neighboring states. Princeton was a favorite college for sons of wealthy planters, and retained deep links to the South and Southern culture well into the 20th century (recall that Woodrow Wilson, a notoriously pro-Southern Democrat, was president of the university around the turn of the century)... in contrast to those dour Yankees at Yale and Harvard in particular.

New Jersey was pretty split politically in the late 1850s. In a somewhat ironic twist to the Civil War, the northern counties tended to be more Democratic, and the southern counties more Republican. In 1860, the New Jersey state constitution allowed for voters to elect individual electors in the presidential race, so New Jersey's delegation was split, giving 4 votes to Lincoln and 3 to Stephen Douglas--though since the election was held by district, it's worth noting that Democratic electors actually received more votes overall statewide. New Jersey was thus the only free state not to pledge all its electors to Lincoln in the 1860 election. Charles Olden, the Republican governor, was elected on a very scant margin the year before.

The 1862 midterms went relatively poorly for Republicans in most northern States (though Republicans still held a majority in most States; they were simply weaker majorities), but perhaps they went most poorly in New Jersey. Olden was replaced as governor by Democrat Joel Parker, a stalwart anti-Lincolnite. Parker criticized the Lincoln administration at every turn, from Lincoln's suspension of habeas corpus to the "outrage" of the Emancipation Proclamation (though it's worth noting here that Olden was not exactly enthusiastic about the Proclamation, either). Parker fostered a Copperhead position and supported peace overtures to the Confederacy, despite opposition from politically active supporters of the Union Army. Parker, of course, was no idiot though. He recognized how lucrative war enterprises were for the State in producing goods for the Union Army, so he stopped short of outright support for the Confederacy.

In 1864, there was another fairly poor showing for Republicans, as New Jersey was one of only 3 Northern or Reconstructed States to go for McClellan. It also seems that New Jersey altered its policy toward choosing electors, though I will be honest that I cannot source this directly to Copperhead influence on the legislature between 1863 and 1864 (but I suspect that may be the cause). The stubborn Democratic legislature insisted on rejecting the 13th Amendment, joining only Kentucky and Delaware among the Union States that initially rejected it. New Jersey Democrats would also go on to initially reject the 14th and 15th Amendments as well, maintaining solid opposition to Reconstruction policies. And like many Southern States, New Jersey maintained or raised its poll taxes as a way of discouraging even the relatively small black population from voting in the State.