Rumors held that as many as sixty Barbary men-of-war were actively prowling the English Channel, waiting for the opportunity to capture more product for the slave markets of Algiers and Tripoli. For most of the seventeenth century, an English or Irish family living near the coast confronted the real possibility that the might be hauled off without warning….[the] numbers suggest that the odds of sudden enslavement by Barbary pirates were far higher for the average Devonshire resident than the odds of experiencing a terrorist attack in a modern-day Western City.
From Steven Johnson’s "Enemy of All Mankind."
Much of what I discuss below comes from Nabil Matar, Britain and Barbary: 1589-1689 (University Press of Florida, 2005). It's an excellent source, albeit a bit dry, and I would recommend reading chapter II in its entirety.
One could say that there are three reasons why the English "couldn't" stop the Barbary corsairs: tactics, economics, and politics. I'll talk about each in turn.
By "tactics" I mean the general difficulty of effecting any maritime capture, especially against irregular (non-military) ships. You had to find the ship, catch the sip, and subsequently seize/destroy the ship. Those were difficult tasks, especially on the open ocean. Initially, the English simply didn't have a large enough navy to secure every merchant ship or coastal city, and the corsairs exploited these significant gaps in England's security. Note that these tactical difficulties didn't really go away until the 20th century with the invention of radar and radio communication, which allowed smaller navies to police greater areas more effectively. And even today it is hard to track/capture small craft. Suffice it to say that no matter how great the navy, there will always be some level of maritime crime that will slip under the radar.
These tactical issues weren't unique to this time period, or region, or set of actors. Let's get down to specifics. To understand the economic and political factors, we need to discuss how the Barbary corsairs operated. First, they weren't really "pirates" in the legal or cultural sense. What they were doing wasn't really illegal as far as the North African rulers were concerned. The corsairs were more like American and European slavers than pirates, although Barbary slavery was very different from North/South American chattel slavery...but that's beyond the scope of this post. People captured by corsairs were taken to a Barbary port city, where they were either held captive or sold into slavery. Importantly, most captives/slaves could be freed if a ransom was paid, which became the primary way that England sought to repatriate the "thousands" of people taken in seventeenth century.
I should point out that although folks talk about the "Barbary pirates" as though they belonged to a single state, North Africa was not homogeneous. It was divided into independent states like Morocco, Ottoman regencies like Tunisia and Algeria, and independent city-states like Salé. Diplomatic solutions were therefore inherently limited unless one negotiated with all governments simultaneously, because corsairs could simply take their cargo to a different friendly port.
According to Matar, the corsairs first began to attack in the English Channel in 1625, the year Charles I ascended the throne. After only a few Englishmen had been captured, Charles sent an envoy to Mulay Zaidan, the Moroccan potentate, asking that he do something to return the captives. Zaidan agreed to ban the sale of English citizens in Moroccan slave markets. But this ban proved ineffective: not only could the corsairs take captives to other ports (most notoriously Salé, a city-state on the Moroccan coast outside Zaidan's reach), but Zaidan's influence within his borders could not overcome the local/regional protections enjoyed by most slavers. Attacks became more frequent toward the end of the decade and into the next, indicating that the diplomatic route was a failure.
Charles I came under increasing pressure from merchants and politicians in England to stop the Barbary raids, and he was initially reluctant to do so for economic reasons. First, war was incredibly expensive, perhaps even more expensive than ransoming the captives (I'll talk about this more below). Second, Charles I apparently believed that fighting the Barbary corsairs would jeopardize the incredibly lucrative trade routes that came through North Africa and the Ottoman Empire. Taken together, the expense of war and the loss of trade would likely exceed the amount lost to the corsairs through theft and ransom. And so in the decade following his ascension in 1625, the King did practically nothing to stop the attacks.
But pressure continued to mount. As more Englishmen were captured, more ransom money had to be paid to the corsairs. Enriching his Barbary enemies only strengthened them, and by 1636 the king's advisors pushed him to do something more. In 1637 he allied with Morocco to attack Salé, which by then was hub of corsair activity, and attacked the city with the full might of the English navy. The operation was a success. Salé was returned to allied hands, and the fleet sailed back to England with ~400 captives. A much needed victory.
But it was short lived. The attack on Salé did little to quell the corsairs sailing out of Algeria and Tunisia. To fight against the "fleet of 60 sail ships" that Charles claimed still roamed the waters, he would need a more robust navy. So why not build one? Here we come to the political side of things. Two important things to note about Charles I: he was king of England, Scotland, and Ireland, which were separate countries; and he believed in the absolute, supreme, and divine power of the monarch. From 1629-1640, he tried to rule England without parliament. This limited his ability to levy new taxes, which in turn limited his ability to finance a new navy. He did try to do so through Ship Money, which is a kind of tax that can be levied by sovereign prerogative without parliament, but this was ineffective and extremely controversial. In essence, the corsairs were allowed to flourish because Charles refused to cooperate with parliament.
Things changed in 1640. Charles reconvened parliament to help with a number of matters, including financing the first conflict in what are known as the Wars of Three Kingdoms (i.e., England, Ireland, and Scotland). In addition to that war, Charles asked parliament to finance a navy to fight the corsairs, but they refused. According to Matar, "The Commons refused to 'concur' [with his requests for 'Ship Money'] unless the king addressed their grievances about church discipline, property and parliamentary liberty." (62) Charles used this refusal as a pretext to dissolve the parliament after only 3 weeks in session.
Now, the Wars of Three Kingdoms is a massive topic, and I'm not prepared to write about it in detail. But suffice it to say that from 1639-1653, various parts of the British Isles were embroiled in conflict (including three civil wars in England!). These conflicts served as a distraction that allowed the Barbary corsairs increased opportunity to plunder English, Scottish, and Irish ships and communities. But Parliament did try to stem the bleeding. In 1641, the Commons passed a resolution authorizing military action against Algiers, although nothing happened. In 1642 the parliament admitted defeat, and began levying a tax that would be used to pay ransom money from English captives. This was the strategy parliament pursued until the execution of Charles I in 1649.
From here things become very complicated. Cromwell recognizes the need for a vastly expanded navy to deal with the corsairs, and one is built (starting in 1650). From there, military actions increase which push the corsairs back away from the British Isles. Long story short, England becomes a dominant force in the Mediterranean, suppressing Barbary activity to some degree. Eventually the Barbary states change tactics, and begin charging European and American traders protection money. England gets imperial eyes for North Africa, Charles II comes into power, and the rest is history.
In sum: the English "couldn't" stop the Barbary corsairs because of internal power struggles, various wars and civil wars, the need to maintain lucrative trade routes, and the general difficulty of suppressing small-ship, irregular actions at sea.
Finally, small complaint about Johnson: that portion of the book is making a remarkably sloppy argument about the comparisons between piracy and modern terrorism (which is the subject of my dissertation...). The comparison he draws at the end of the quotation is baffling, since the risk of terrorism in a "modern-day Western City" is and always has been extremely low, and the War on Terror isn't only about securing "modern-day Western Cities." Although his analysis and argument are going in the right direction, his equivocation is misleading.
Because the English Navy simply wasn't up to the task - it had declined significantly from recent Elizabethan heights.
One major problem faced by 17th century states was the increasing complexity of warfare, and the resultant difficulties in constructing and maintaining fleets. Cannon and the ships they were placed on were extremely expensive, and all European states that sought to maintain a contemporary navy faced growing pains.
One of the defining features of King James's (reigned 1603-1625) reign was the terrible state of English crown finances. The state was in such a large amount of debt that James' first parliament literally couldn't comprehend the number, and assumed it was ten times as large as it was. But not just uncomprehending, Parliament was also *unwilling* to pay enough taxes to make the crown financially soluble. They argued that if crown debt was so high, it must be a problem with spending that taxes alone would never fix. They weren't entirely wrong - James loved expensive parties and gifts for friends - but they also failed to appreciate how expensive an early modern state was to run.
This meant that during James' reign, there really wasn't enough money to invest in the navy. Worse, the money that did reach the navy was filtered through layers of grift and corruption. It was common for naval officers to pocket most of the pay that was due to their sailors, provisioners bought cheap already-rotten meat and supplies and pocketed the difference, ropes had cords removed and re-sold, etc. The navy, so far as many of its officials were concerned, was an opportunity to make money, not actually engage in naval warfare.
Enter the Duke of Buckingham. Starting his career as an attractive boytoy flaunted in front of evidently-bisexual James (yes, really), Buckingham rose to be the most influential man in Britain through controlling access to the king. Although he owed his position to James' affection for him, Buckingham sought to make himself practically useful through whatever means were available to him. Aware of the dismal state of the Jacobean navy, he sought to reform it, and was given the authority to do so.
There were three problems with Buckingham's efforts to reform the navy. Firstly, despite what he thought of himself, Buckingham was no saviour of the Protestant faith who was going to lead the English fleet to abstract victory. He had no real talent or experience in naval affairs. Second, despite Buckingham's efforts, there was still a lot of corruption. And third, there were political difficulties which meant that the navy's deployments in 1625 and 1627 to Cadiz (a major Spanish port) and La Rochelle were underfunded, and hence food and supplies utterly inadequate to the task; of the 7000 English deployed in the amphibious La Rochelle campaign, just 2000 survived.
So ~1600-1627, the English navy is not so good. But things got better.
During Charles' personal rule (1629-1640), the Crown sought out alternative revenue raising streams, since Charles did not have a functional working relationship with Parliament. The most controversial of these was shipmoney, which was an extraordinary tax on coastal counties used to pay for ships. As you point out, the English had legitimate fear of pirates in the 17th century - they were also constantly bothered by Catholic Dunkirkers, so the first round or two of shipmoney were palatable to the taxpayers, since they did place ships in the water protecting English interests.
Shipmoney soon became unacceptable to parliament, however. King Charles demanded shipmoney from *all* counties, not just those that were coastal, which was not the done thing. And he asked for it regularly, even after new ships were seen to adequately protect English naval interests, even re-imposing licenses on Dutch fishing off British coasts.
As all taxes were *supposed* to be approved by parliament, shipmoney was unacceptable. So unacceptable, in fact, that it was one of the first political issues that people standing for English parliament ever campaigned on. When Charles was finally forced to summon a parliament in 1640, after 11 years of personal rule, the first thing they wanted to talk about was shipmoney. Unaddressed in the so-called Short Parliament, political tensions flared up even higher into the English Civil War.
The English navy was too busy fighting the English Civil War to contend with barbary pirates from 1642-1647 or so; Irish pirates increased their activity. After the Parliamentarians won the war, there was significant naval reform in which several naval commissioners shared the top job, although Robert Blake (sometimes regarded as the founder of the modern British navy) was the most important person to win the job.
One of Blake's most important reforms was in naval tactics. Although it had been English doctrine to build swift ships with cannon for some time now, most captains preferred to board their enemies, as this meant they could loot valuables rather than sink them. Over the course of the Anglo-Dutch war, Blake had most of his captains fighting at range with guns; part of this success was the new cash prize system, which meant sinking enemies was not the agonizing financial loss it would have been otherwise.
Over the next several years, Blake worked to improve the diplomatic situation of the Commonwealth through force: obliging Portugal and Spain to accept the Commonwealth, and defeating the Dutch in the first Anglo-Dutch war. When the navy was finally a free agent again in 1655, Blake set out to attack the Barbary pirates. He attacked the barbary port of Tunis, and successfully destroyed the ships in harbour and the shore defenses - the first time ships had ever done so unsupported by land forces. This wasn't nearly as decisive as the English would have liked, as barbary activity did not cease, but it did prove that the English were now, by mid-century, more than a match for Barbary corsairs.