This question pops up here from time to time. Accuracy, if you think about it, is the gun doing the same thing over and over again, so that it's possible to predict where the ball will strike. In order for the gun to shoot the same way, as much as possible has to be the same: the ball has to come out of the barrel the same way each time , so it has to be a tight fit. Then, to do this the ball had to be patched, or wrapped in a piece of cloth. This would also make it grip the rifling. The powder charge has to be the same each time- then, that meant it had to be measured. Importantly, if measuring the charge and patching the ball could not be done, a rifled bore would not help the accuracy. Measuring the powder, using a patched ball, meant a rifle was slower to load and fire: a musket could get off as many as four shots in a minute, a rifle perhaps one. Because they loaded more slowly, rifles could not easily stop a charge. One example of this was at the Battle of Long Island, where a number of the militia coming up from Virginia in the famous Beeline March to Cambridge ended up being routed, some stuck to tree-trunks by Hessian bayonets...which was another problem; rifles did not have bayonets.

However, rifles were quite popular in many parts of the colonies, especially Virginia, North Carolina, western Maryland and Pennsylvania. It was not unusual for a hunter to be using the same rifle for years, so some of them were quite good shots, and as they could shoot accurately up to around 300 yards ( though there are records of them accurately shooting further) they were quite handy for skirmishing.

Ironically, though, much of the expertise in making their rifles had come from immigrant German gunsmiths. Germans had excelled at making small hunting rifles, and many German states had their own population of hunters who knew how to use them. When the British brought Hessian troops, they came with Jaegerkorps. And at the same Battle of Long Island, when the colonial militiamen with rifles were routed by Hessian soldiers with muskets, there was also successful skirmishing by Hessian riflemen.

By the Napoleonic Wars the British had their own rifle units ( thus the popular series Sharpes' Rifles.) Why they did not have them earlier I can't answer. Certainly, there were fine British rifles made in the 18th c., and breech-loading ones invented by British Major James Ferguson were actually given to a unit of a hundred soldiers for special trial, and might possibly have been used at the Battle of King's Mountain ( a controversial question). But the rifle was deemed too expensive.

Eventually it became possible for a rifle to be loaded as fast as a musket without sacrificing accuracy, with the invention of the percussion cap circa 1840 and then the invention of the Mine' bullet circa 1850.

The adoption of the rifle into the British Army took a considerable amount of time due to entrenched resistance by existing military authorities, primarily because it was inferior to the musket in direct short-range firepower- as r/Bodark43 says, the rifle could put out between a third or a quarter of the shots of a musket (depending on source) due to more difficult loading procedures- this was an era where infantry battles were decided by volume of shot and force of charge, and British Army doctrine in particular focused on drilling men in maximising quantity of firepower output. The first British unit to be equipped with Baker rifles was the Experimental Corps of Riflemen in 1800, renamed the Rifle Corps, being brought into the line as the 95th Regiment of Foot (Rifles) in 1803. Crucially, the unit did not operate as a conventional line regiment- it neglected principles of formation drill and hierarchical observance, with all ranks wearing green rather than red coats as prototype camouflage and focusing on accuracy rather than speed of reload time. When the regiment deployed, such as in the Peninsula Campaign, it is clear that it was consciously used by commanders as a special unit for skirmishing and harassment, in contrast to both line infantry regiments and even the light companies of those regiments. Source: All the King's Men: The British Redcoat in the Age of Sword and Musket, by Saul David.