So I notice that this question is getting some traction, but has no responses. This is probably because a good answer to this question would require a very long reply. Roman public finance is a huge topic and goes through a lot of changes in the long centuries of Roman history. What I will do is write a shorter, place-holder answer to this and hope that someone else has the time to write something more substantial, because it is a great topic.

In the Republic, the public treasury was called the aerarium. It was administered by public officials like the quaestors and the praetors (among others) and was fed by taxes, primarily from provincial taxation but also from things like revenue from public land in Italy, port-related taxes, and utility access bills. Augustus, when he established the principate, reorganized public finances along with everything else. He divided the administration of the provinces between "Imperial" areas and those areas which would remain under the governance of Roman proconsuls. The revenues from the "Senatorial" provinces continued to feed the aerarium, but the revenue from the newly-designated "Imperial" provinces (including Gaul, Spain, and Syria) would instead funnel into a new fund, called the fiscus (which means "basket" or "little wallet" in Latin). There was also a third public treasury created in 6 CE, the aerarium militare This one was responsible for paying soldiers, and was fed by "private" donations from Augustus and also by a new 5% tax on inheritances.

The aerarium under the early Empire continued much as it had in the Republic, fed now by revenue from the quiet (and thus usually lucrative) "Senatorial" provinces. Technically it was administered by the senatorial officials (quaestors, etc), but Emperors sometimes meddled in it or moved funds in or out of it (I don't have a specific example at hand at the moment). The fiscus was always directly in control of the Emperor, and included both the revenue from those "Imperial" provinces and also the Imperial private fortune, which was considerable even in the time of Augustus and continued to grow through the generations. The history of this word is a little complicated. In the Republic, fiscus could refer to any person's private wealth. During the early Empire, there was immediate confusion between the Emperor's fiscus and the Fiscus with a capital F, that fund which received the revenue from Imperial provinces. The two are conflated in the sources and must have often been conflated at the time. It is probable that the distinction was never very important: the Emperor was the state, and his "private" money was the state's money, particularly when it came to the Fiscus with a capital F. More properly, the Imperial family's "private" wealth was the patrimonium, eventually called the patrimonium caesaris. This was the technical "fortune" of the Emperors which was passed down (via the machinery of inheritance, for the most part) from Emperor to Emperor (often via formal adoption). There was also the res privata principis ("private assets of the princeps") and I admit that I'm not entirely clear on the nuances between all these.

By the time of Hadrian or perhaps Septimius Severus, even the distinction between the Fiscus and the aerarium had begun to fade, with the aerarium increasingly losing its identity. By the late Imperial period, the two terms are used indiscriminately.

Now, as far as OP's question specifically: the fiscus, because it was fed by provincial revenue, was no small fund. It was huge, and from the very beginning it was confused with the res privata and/or patrimonium of Augustus (and then the Julio-Claudians, and then the Flavians, et cetera). The fiscus wasn't just a big pile of money for the Emperor, however; there were certain demands on it related to the running of the state, and in fact there was a complicated web between all the flavors of funds in the Empire, and an equally complicated army of elected and appointed administers to those funds.

A great place to read more about this topic is the new(ish) book put out by Erdkamp et al, called Ownership and Exploitation of Land and Natural Resources in the Roman World (Oxford 2015). In it, there is a chapter "The Imperial Property and its Development" by Lo Cascio which probably has some great insight (I haven't read this yet myself and don't have the book handy).