Just four years after the 1841 census, the Great Famine struck Ireland incredibly hard. Most of the rural Irish population farmed for subsistence, and the staple crop was potatoes. The Blight caused the 1846 harvest to take a hit, and in subsequent years made almost all potato crops rot. Although numbers are not documented well, it is estimated that throughout the famine, one million died from starvation alone. Another two million would emigrate within the decade. Ireland would still experience the after effects of the famine for decades. Emigration was still much higher than it was pre-famine, and birth rates were down in the country. All of these factors led to the large population decline from the 1841 census to the mid 20th century.

Sources: https://www.britannica.com/event/Great-Famine-Irish-history

http://www.mapspictures.com/ireland/history/ireland_population.php

http://www.newworldencyclopedia.org/entry/Irish_Potato_Famine_(1845–1849)#Emigration

Keep in mind that "Ireland" means a couple different things and that may be a part of what is confusing you.

In addition to millions dead in the famine and millions more emigrating, Ireland was partitioned in the 1920s into the Irish Free State (now officially known as the Republic of Ireland) and Northern Ireland, which remained a part of the UK. The 1841 census covered the entirety of the island of Ireland, where over 8 million people lived. However, 1.6 million or so people lived in the six counties which would become Northern Ireland.

Fast forward to 1960, and that 3 million figure is just for the Republic of Ireland, which means it doesn't count the 1.2 million or so people living in Northern Ireland.

So the drop was from 8 million to 4 million if you count the whole island, or 6.5 million to under 3 million if you count just the area now known as the Republic of Ireland.

This is still obviously a huge drop, and yes, the famine had a large part in it. Ever after people stopped dying in large numbers, millions more continued to emigrate to find better opportunities.

EDIT: Should not have used the term "officially known as the Republic of Ireland." As u/ramak__ points out, it is the official "description" of the country, and a useful one when differentiating between the two countries, but the official name of the country is "Ireland" (or in the Irish language "Éire").