Fletching is the main theory I've always heard. Feathers ( go take a look at a wing feather) have a curve to them. If you have all the curves going in the same direction, when you fletch an arrow, the arrow will spin and fly straighter. So, all that was necessary to figure that out was to mix up the curves on one arrow, align the curves on another, and see which shot straighter.

Another theory is that it was devised as a solution to powder fouling. Early black powder was even messier than the present stuff : it took a while to realize how to make it well, so early powder was often not well-mixed, which resulted in more solids left in the bore after every shot. Grooves cut into the bore would in theory allow fouling to be shoved out of the way, allowing the gun to be loaded more times before it needed to be cleaned ( always a nice thing, if more people keep running at you with lances...) In support of this theory, the earliest rifles , from southern Germany circa 1460, had eight to ten wide grooves, parallel: not spiraled. But these may not have been the norm...the Turin Armory listed a firearm with a spiral grooved barrel in 1476. That's not much later, so it could just be a matter of what records have survived.

You would think that spiral grooved barrel would shoot better than a straight grooved barrel and, like the fletchers, gunsmiths would then have quickly figured out which one to do. But it may not have been so simple. Accuracy is the same thing as consistency- the gun doing the same thing, over and over, so that the soldier can use the sights to predict where the next bullet will go. Given the variability in gunpowder at the time, there may have been little difference between spiral-grooved and straight-grooved, because there would be variations in the powder charge. Also, modern conical bullets need to spin to be stable- the longer they are, generally the faster they need to spin to avoid tumbling. Round balls need very little spin: really just enough so that they are turning on the axis of the bore of the gun/ the flight path- if they don't, unequal air pressure will cause them to drift. So, the twist of early rifles can be quite slow, can be a very long spiral.

Thorsten Lenk: The Flintlock and its Development