The thing to keep in mind about many fantasy novels is that, geographically and demographically, they don't compare well to the real world. Tolkien's Middle-Earth is full of vast empty spaces, places where people plain don't live. While western Europe's population decreased in the period after the fall of Rome (before rebounding in the 12th and 13th centuries, dipping in the 14th, and rebounding again in the 15th-16th), one would be hard pressed to find large areas that were completely devoid of population. When there are fewer people in a region, they spread out, they don't generally cluster into tiny, crowded enclaves.

It is true that trade decreased in the post-Roman period. But it would be folly to think it disappeared entirely. Blades made in the Frankish Rhineland made it to Scandinavia; coin looted by the Vikings from England have been found in Baghdad. More than that, areas of human habitation were not so widely dispersed that there wouldn't be any travel between villages and farms, or villages and market towns, or cities and cities. By 1100, we see definite evidence of the renewal of trade on a large scale, even in northern Europe, with English wool traded in the cities of Flanders, while Aquitanian wine dominated the wine trade in England.

During the medieval era, French population reached a highest around 20M inhabitants before the black death (early 14th century), which represents roughly the population density of the modern US or of the state of Missouri; it's lowest population was around mid to late 9th century with some 6M inhabitants, half what it had before the germanic invasions but still a density equivalent to modern Nevada.
France was, at the time, the most densely populated place in Europe with Northern Italy, and as you can see with these data, that is quite an higher population density that Tolkien described in LotR.

Source : Jacques Dupaquier, Histoire de la population française