I know that feudalism came from Manoralism, and that villages essentially sprung up around fiefs as communal housing for the serfs under the lord.

There's a whole debate that could fill this entire sub as to the extent to which feudalism is really 'a thing', with the broad consensus being that is isn't. It's more of a convenient umbrella term that we project to encompass a whole range of interpersonal relationships, hierarchies and methods of economic exchange. Certainly, as you said, a manorial system is at the centre of what is largely an agrarian economy. The development of a settlement can hinge on a wide variety of details, one of which is pure luck. In England, for example, a significant number of later medieval cities develop from the sites of the 9th century burghal network, but some become burhs because they are already major settlements - like Hamwic (modern Southampton) or Exeter - and just as many burhs don't develop into major settlements, or do, but at a later date for reasons unrelated (Runcorn is a prime example).

A trade network, especially maritime, is indeed often key: Bristol and Chester come to economic prominence in the Tenth Century due to the rapid proliferation of trade with Hiberno-Norse settlements across the Irish sea, as well as across the Bristol channel and with North Wales and the Scottish Islands respectively. Sometimes, a particularly valuable resource and/or sheer luck make a settlement particularly viable: the largely chance discovery of enormous quantities of silver on the royal hunting grounds at Carlisle in the 1090s led to the extraordinarily rapid growth of a boom town which by the 1170s was described by Jordan de Fantosme as boasting some of the finest fortifications in England and was host to several booming international trade quarters, in an area that had previously been essentially desolate.

Many settlements change over time, especially as a result of the huge economic and demographic changes of the industrial revolution, and as a result their previous status is lost to a modern eye. Lichfield, for example, is today a small town in the Birmingham/Stoke-on-Trent commuter belt, but in the Eleventh Century was an important clerical centre, the seat of a major bishopric, and situated conveniently both on the Roman Watling Street, and adjacent to the former Mercian capital at Tamworth. Similarly, Wimborne Minster in Dorset is today a sleepy market town visited mostly by tourists and commuters to Poole and Bournemouth, but during the medieval period was a fairly substantial rural settlement (among the biggest 30% recorded at Domesday) and a major centre for the wool trade, with a royal manor and an important minster church.