Wikipedia shows a population decline of 30,000 people in the this timeframe, including a weird demographic curve, but the history section of the article does not appear to explain the population decline. The other Canadian maritime provinces did not experience the same demographic curve. It looks like the population did not recover to 1891 levels until after 1966. What factors contributed to this population decline?
Prince Edward Island was ill equipped for the industrial age. It lacked coal and water resources essential to industrial development, and the cost and availability of transportation proved to be a difficulty that has not been completely overcome. Industries on the Island were soon crushed by larger and more efficient plants in central Canada, but at the same time the national policy provided no protection or markets for the Island's natural products. The change in technology was most strongly felt in the shipbuilding industry. As the wooden sailing ship was replaced by steam vessels constructed of iron and steel, the entire industry died, having neither the raw materials nor the capital to make such a fundamental shift.
In the 1890s an economic depression, or slowdown, hit the province hard, and many of Prince Edward Island's residents moved to western and central Canada, and the United States, where employment opportunities were more plentiful. Boston, Massachusetts, was the destination for most people who left the Island during this period.
Source: The Canadian Encyclopedia: Year 2000 Edition [James H. Marsh]