The problem was with the Bank itself. You can't compare this bank to the modern Federal Reserve system, because the US Government controls the modern Fed completely, and the US Government only owned 20% of the Second Bank of the United States. The bank was even more powerful than the current Fed in some ways though. It's directors were able to control the flow of capital throughout the country, and could reward and punish their political friends and enemies as a result. They were also making a huge amount of money off of their investment. From Jackson's POV the Federal Government was enabling a small cabal of super rich citizens to exercise so much influence over the rest of the country the amount of power they had was in and of itself corrupting and undemocratic. Jackson wanted to eliminate the Second National Bank and form a new Federal Bank as part of the Treasury Department.

Here is a link to Jackson's letter to Congress explaining his veto of the renewal of the bank's charter: http://avalon.law.yale.edu/19th_century/ajveto01.asp

This letter is a bit dense for a modern reader, but well worth reading. It is one of the most important documents in American history, close to the Declaration of Independence, Bill of Rights and Emancipation Proclamation in importance. This was the turning point where the American Government ceased to be the enterprise of the rich and powerful only, and began it's journey to a truly representative democracy of a wide electorate. The Jackson Presidency is a watershed because of his fight to open democracy to the masses, and the fight over the bank charter renewal was where this all played out.