Why did the economy remain mostly agrarian and not modernize with manufacturing?
Your two questions aren't quite the same. I'm not sure what you mean by "die down," but the economy of the most regions of the Ottoman Empire did not suffer under Ottoman rule, in fact in many places it grew. The exceptions were mainly the frontier zones of Hungary, bordering the Habsburgs, and eastern Anatolia, bordering Persia, which were worn down by frequent wars.
On the second question, about remaining mostly agrarian and not modernizing - I'm not sure this is the right way to pose the question, because this is true of almost all regions of the world. The question that needs answering is why the Industrial Revolution did happen in northwest Europe, not why it didn't happen everywhere else. Once the technologies of the industrial revolution began to spread, the Ottoman Empire was not slow to adopt them. In fact, it adopted them much sooner than many parts of the world, which is not surprising since it was close to Europe, a significant trading partner, and many western European capitalists had investments there. However, as with most of the world outside western Europe, North America and Japan, the economic development in the 19th-century Ottoman Empire were mostly focused on natural resources and cash-crop agriculture rather than higher-value manufacturing.
This points to another complication in the story. It's not really the case that the Ottoman Empire was agrarian and simply remained that way. In fact, it saw significant deindustrialization during the 19th century, with traditional manufacturing collapsing and the share of labour involved in agriculture increasing. Although, like all pre-industrial revolution countries, the early modern Ottoman economy was primarily agrarian, it had a relatively large manufacturing sector by pre-industrial standards. It had a significant textile manufacturing sector, based in western Anatolia, Syria and parts of the Balkans. Along with the Chinese and Indian textile sectors, this was pretty much wiped out during the first half of the 19th century by British competition.
Why did this happen? Primarily (and similarly to China and India), because the Ottoman Empire was politically weak in the 19th century, and so was manuevered into a trading relationship that suited British industrial interests. The Anglo-Ottoman Commercial Convention of 1838 (which was basically the price for having the British intervene to stop Mehmed Ali, the rebellious Ottoman governor of Egypt, from invading Constantinople) ensured low tariffs on British textiles, so there was no way Ottoman manufacturers could compete. By contrast, the rising industrial powers like Germany and the USA maintained protectionist tariffs throughout the 19th century, precisely to stop Britain, which had first-mover advantage, from flooding their markets with cheap goods and preventing their own manufacturers from developing. And Britain itself, of course, had had protectionist policies before it came to dominate the world economy, at which point it switched to promoting free trade.
There are some other reasons as well. Landlords were a powerful lobby in key parts of the empire, and their interests were suited by economic policies that favored cash crop agriculture to supply western manufacturing industries (cotton, sugar, tobacco, opium, etc.) But I think the political imbalance, and the inability to impose protectionist tariffs, is the key reason why Ottoman manufacturing was hampered, as was true of manufacturing across Asia.