I've read that George Washington set a large precedent with his peaceful transfer of power to John Adams. Were people at the time actually concerned he might try to hold on to power, possibly through violence?

@OP, do you possibly mean John Adam's peaceful transition with Thomas Jefferson? Adams was essentially Washington's successor, and operated within the same faction. It would anachronistic to call them a party at the time, but the friends of the government (Federalists) supported Adams and followed in Washington's footsteps. It was the ascent of Jefferson's Republicans (no relation to the modern party) that represented a real break in governance.