There is almost certainly a population decline in the third century - as you probably know it was a time of immense difficulty for the empire, with war, famine, and plague.

The fourth century, however, is now considered a time of recovery, both economically and in terms of population. There has been a lot of talk about climatological events having substantial impact on this see-saw, and I believe Kyle Harper at the University of Oklahoma has something in the publishing pipeline on this. He recently gave a lecture which incorporated it, "Population, Disease, and the Fall of the Roman Empire: a Biohistory" in The Encounter of Science and History: a Conference on the Science of the Human Past held at Harvard in November 2013. Similar lines have been suggested in:

• Brown, Peter. Through the Eye of a Needle: Wealth, the Fall of Rome, and the Making of Christianity in the West, 350-550 AD. Princeton, N.J: Princeton University Press, 2012.

• McCormick, Michael. Origins of the European Economy: Communications and Commerce A.D. 300-900. Cambridge, U.K. ; New York: Cambridge University Press, 2001.

• McCormick, Michael, Ulf Büntgen, Mark A. Cane, Edward R. Cook, Kyle Harper, Peter Huybers, Thomas Litt, et al. “Climate Change during and after the Roman Empire: Reconstructing the Past from Scientific and Historical Evidence.” Journal of Interdisciplinary History 43, no. 2 (July 27, 2012): 169–220. doi:10.1162/JINH_a_00379.

Whether this leads to a net population +/- over the two centuries combined, I don't know and I haven't seen a consensus, but I believe the data are vague enough that either guess would be within the margin of error.