"Would the USN have been able to challenge the Royal Navy [in 1861]?"

Not. At. All.

I'm sure a 19th century British historian on this site could chime in with a more detailed answer, but to be succinct, the USN in 1861 couldn't have held off a determined British Naval attack for more than an hour or so. The British navy at this time was one of (if not THE) strongest in the world, and with men who were seasoned, well-trained, and full-time professionals. At the beginning of the US Civil War, the Union had roughly 40 ships in active service.

40

They had another 45-50 that could be pressed into service pending the enlisting and training of crews, but even so, this number is laughably small. These vessels were scattered all over the place, as well, so it was no small feat getting them all organized and positioned to maximize their use. They were also a hodge-podge of styles (some weren't steam vessels, but strictly sail, which made them mostly obsolete), and some would have only been good for guard duty, so to speak.

Now, by the end of the war, the Union had something like 670 vessels, which was a massive improvement, but even so, they could have never hoped to seriously compete with the British. I'd defer to a British historian on the exact number of British vessels in service during the ACW, but I'm confident that it always outnumbered the American vessels, both in number, quality, and gun displacement.

[Sources: Tom Chaffin, 'Sea of Grey'; James McPherson, 'Battle Cry of Freedom'; Shelby Foote, 'The Civil War, vol. 1']