I get that "fugitive slave act BAD" but personally I think it was a necessary evil to pass the compromise.

That's just not so at all. There was a point when the plan was to put all the measures for the capitulation to slavery of 1850 into one bill, so everyone would have to vote for what they hated to get what they wanted. Henry Clay thought it would work and he'd been around this block a few times, albeit without trying an omnibus route. The original omnibus bill, reported out of his Committee of Thirteen on May 8, left out the Fugitive Slave Act and the District of Columbia slave trade as separate items. So already the rest is not contingent upon the FSA, which I must remind you was no more than a bill to legalize kidnapping and reward special judges if they found in favor of it rather than freedom in a fake trial where the accused was not permitted to speak in their own defense. What finally passed the Senate went out like this, with the House bills in the same sequence but a bit later:

Date	Bill
July 31	Utah Territory
August 9	Texas Border Adjustment & Debt
August 13	California Statehood
August 15	New Mexico Territory
August 23	Fugitive Slave Act
September 16	Abolition of public slave trading in the District of Columbia

The House does pass an omnibus which the Senate later accepts, but it only includes California's statehood and the Texas adjustments, not the Fugitive Slave Act.

One could argue that some vote trading went on anyways, but agreements between senators to vote one way or the other on bills are essentially pinky swears. Unless it's all in one bill, there's no way to ensure that someone doesn't flip on you. The best one could do is ensure that the other side votes for their poison pill before you do your own, which the timeline does not support. If Southerners wanted to make their votes for California's statehood, for example, contingent on the Northerners voting for the Fugitive Slave Act, they ought to have insisted on voting for the FSA first. They did not because the Fugitive Slave Act was not a major priority for most of the South, being instead a special interest of the Border South supported in substantial part out of sectional solidarity. They were far more concerned about the ability to enslave people in New Mexico and Utah, which they secured.

But let's leave that aside. Fillmore was a nonentity while most of this went down because he was the Vice-President and nineteenth century vice-presidents chiefly had the job of not dying in office, which a few could not manage. Fillmore did and he comes in at the end, after Taylor has some bad cherries. With Old Rough and Ready six feet under, he's the president and the Whig Party leader. This all means he controls the levels of party power, chiefly the patronage. So he can buy votes by offering offices to politicians and their friends. Taylor could have done that too, but his own ideas on the compromise were not the same as Clay's or anyone else's. His big idea was to immediately admit California and New Mexico as states without saying anything either way about enslaving people. This would duck every controversy by leaving it up to the white people there to decide and not embroil the nation in contention over alleged fugitive slaves or slave trading in the capital. And if the South did not like it, Taylor might just lead an army to ensure their compliance.

Millard Fillmore is not going to do that. Instead he opened his pockets and threw his full weight behind the five now-separate compromise measures, leaning at least as far South as the actual Southerner Taylor tilted North. The resultant voting was not materially very different from prior capitulations to enslaving: each section voted mostly as a bloc for what they wanted and it carried thanks to a small number of politicians who colored outside the usual lines. A few years later, Sam Houston bragged during the next slavery controversy that he was only one of very few men in the US Senate who voted for the whole slate. Even Stephen Douglas, who ended up managing the passage of the bills, couldn't say that. (He abstained on the FSA.)

This doesn't make Fillmore the sole villain of a non-compromise which settled the Slavery Question for a much-ballyhooed "forever" of less than four years, but he's a major operator once he becomes president and he is both the guy who signs the bills into law and the one who goes about very aggressively prosecuting the FSA, to the point where he tries to get about twenty Pennsylvanians arraigned on charges of treason for resisting a fugitive rendition and, in 1854, employs military force to see a rendition completed in Boston. The white North, including many men who hardly considered themselves antislavery, the FSA was so utterly odious that they were happy to declare in their papers and public meetings that they would render it a dead letter. Helping fugitive enslaved people may well have been the most popular form of antislavery activism, usually done in a fairly public manner for decades prior. Making that a crime which results in fines and perhaps jail, and then also making slave catching effectively a draft into which any white person might be taken on the whim of a slave catcher, is several bridges too far and at a stroke validates the correct view that the South will not be content until the free labor society of the North is utterly crushed. That was the kind of thing only antislavery radicals often believed before 1850. This and other capitulations across the decade make an excellent case for a purpose-built antislavery party, which emerges from the wreckage of Northern Whiggery and the split of the Democracy to elect a president in 1860.

Attempted neutrality and people-pleasing were not Fillmore's flaws. (A case for the latter may be made for Franklin Pierce, but I think it's overstated there too.) He knew what he was about as well as anyone did. He understood his job as arranging a capitulation to enslaving, to preserving a union built on it against all threats to it. This does not meaningfully distinguish him from anyone else who occupied the Presidency before 1861. If he wanted neutrality he could have stood on the prior Fugitive Slave Act, which was far less draconian. He could have adopted Taylor's No Territory plan to theoretically dodge the slavery issue. He did neither, but was instead an enthusiastic partisan for the latest proslavery compromise. While he was certainly no clairvoyant, he was a Yankee politician who knew the antislavery element of his section as well as any. Other politicians, similarly situated and in favor of the program, understood that there would be a tremendous backlash against all this. All of them and Fillmore himself probably expected it to all subside in time. They underestimated, for once, the complacency of the white, Northern body politic.