Was the Roman province of Britain a net drain on the treasury due to its heavy garrison needs and lack of resources? Was it abandoned in the 5th century because it didn't make financial sense to keep holding it? How robust was the economy of Roman Britain?

So I can speak mostly to the second part of your question, the economy of Roman Britain, specifically in the late Empire, and its strength, or as the case was, the utter lack thereof.

The blow is borrowed from an earlier answer of mine that dealt with the economic situation and changes that Roman Britain was going through, and the eventual effects that this had in the long term.

So assuming for the sake of argument that the Roman troops actually did abandon Britain, itself no sure proposition, what are we going to see as the immediate effects in the near future?

Likely not much truth be told. At this time Roman Britain was a lost cause for Imperial control for a variety of factors and many of the changes that we associate with this time had far longer standing causes and effects.

Economic Changes

For example, the entire region, even extending into Gaul, was in a severe economic decline that predated the collapse of Roman power in western Europe. This made Britain both less attractive for Roman elites to defend and stay in, as well as a drain on badly needed manpower and financial resources. We can measure this economic downturn through a variety of different methods given the lack of historical literary sources that directly record it. Now this is not the same as the Empire deciding to abandon Britain because of its lack of economic profitability. Imperial economics were complex and it is not easy to determine if a part of the empire was "unprofitable" centuries after the empire collapsed. Nor is economic profitability the only determinant in an empire attempting to defend its land and people, issues such as Imperial pride, prestige, militarism, and more also have to be taken into account. Regardless of Britain's economic situation vis a vis the rest of the empire though, it is clear that Roman Britain was in a state of economic decline compared to its heyday in the 3rd century.

One aspect is evidenced by the abandonment of urban areas in Southern Britain, today England. Urban spaces in the Roman Empire needed vast economic support from both the surrounding countryside as well as reliance upon Imperial networks of trade to be sustained. Luxury goods that the urban elite of Britain relied upon such as pottery from Africa, olive oil from Italy, wine from Greece, and so on were no longer available in large quantities by the end of the 4th century and into the 5th. (Now they did not vanish entirely and there is evidence that these trade routes existed in some capacity for as long as a century or two after this period) This specialization of the Empire, allowing different regions to fill in the local resource shortages, was brought to a slow halt as the empire collapsed in the west as a whole, but in Britain the situation was much more rapid and dire. To be blunt, Roman civilization, cities, baths, fortifications, start to fall into disrepair and disuse as the Romanized elite fled from the urban centers to rural villas and new centers of power (ironically these same centers of power were often pre-Roman urban sites such as hill top settlements) that were more defensible in the face of increased hostility across the North Sea world. But this does cut against the idea that Britain was an economic backwater in the empire, it was plugged into the same trade networks that kept Italy, Greece, and the East supplied with goods like African pottery and Egyptian grain. It was only during the tumult and collapse of Imperial trade networks, especially long distance trade, that Britain became cut off from the flow of Imperial goods.

Unable to sustain themselves off of local resources the elite of Britain were faced with a few choices, loose status as Roman figures due to their lack of access to Roman urban life and forge a new living/identity as warlords, retreat to lands still held by Romans, or assimilate into the new populations that were cropping up in lowland areas of Britain.

The urban villa life of the Romanized elite, itself probably a small portion of the population though this is by no means universally accepted, died alongside Roman urban life and the former villa centers became estates for various warlords and newcomers, or were divided up into more manageable settlements. This was obviously quite disastrous for the people who owned these estates, and some clearly anticipated eventually returning as evidenced by coin caches discovered in many of these areas.

However this did not affect the majority of the people actually living in Roman Britain at the time. The overwhelming majority of the population did not live in urban areas, indeed for all the focus on urban sites within the empire, the vast majority of people never set foot in large cities and lived as rural farmers or craftsmen. The collapse of Rome's northern economy did not affect them in their day to day lives. They were subsistence farmers before and remained subsistence farmers after. However they did have to deal with a few changes nonetheless. The most obvious of which were of course the newcomers.

Population Changes

"Germanic" peoples moving into the Roman Empire was not a new phenomena at the end of the empire and there were settled populations that moved relatively freely between the Germanic and Roman worlds with ease. Britain at the end of the empire was already home to many Germanic peoples, sometimes characterized as Angles, Saxons, and Jutes, who had set up service in the area as mercenary fighters, often to see off people who were very similar to them originally. When Roman authority ebbed away in the 5th century, these people were suddenly without pay, and our sources would have us believe that they turned on their British employers and carved out new realms in the land. This was the predominant view of the "Anglo-Saxon migration" to Britain for centuries, and is the version that our two sources from this time, separated by two centuries, record. This is at best an extreme oversimplification.

Robin Fleming argues in Britain After Rome that the immediate aftermath of Roman collapse was an anarchic period that say large populations movement from the low countries, Scandinavia, Northern Germany, but also Ireland and Scotland, into lowland areas of Britain. These communities were not tightly held by any one state or polity and functioned mostly autonomously with a small elite maintaining some semblance of Roman continuity in some fortified areas. Over time these communities slowly coalesced into larger polities that invented a past history of invasion and conquest to legitimize their rule over lowland parts of the island. In most of the island the people who ended up in charge were the descendants of newcomers who migrated from Extra-Roman Europe, namely Scandinavia and the Low Countries, except in the highland regions of modern Wales and Scotland where native rulers were able to maintain legitimacy and in Wales at least, some tenuous connection to Roman identity.

Cultural Changes

Other aspects of life changed too. Christianity famously disappeared from England during this time, only to be resurrected during the evangelizing movements of the 6th-7th centuries. Now while not all Christian communities vanished Christianity did take a back seat during this time and pagan traditions, not entirely removed from those popularized in Scandinavia, arose and came to dominate much of what became England. So this was one major change overall.

Other cultural aspects were slower to change and can only be seen with the benefit of hindsight. This includes changing cultural tastes in various aesthetic choices such as housing, jewelry, and clothing. Many new styles were formed that had their antecedents in both Roman, insular, and Scandinavian traditions. Over time however the cultural focus of much of lowland Britain shifted from the continent to Scandinavia (think the location and figures that dominate Old English epic poetry such as Beowulf, and this would not be undone for centuries, it took the Norman conquest to firmly shake England from the Scandinavia world.

So in short, in the immediate future the changes would be small, especially if you are not a member of the Romanized elite, but these changes will add up over the next two centuries to create an entirely new culture that looked not to Rome for its past, but to Scandinavia and the low countries. This wasn't accomplished through warfare and slaughter, but through migration and cultural assimilation that accompanied the end of the Roman economy in Britain.