I understand that in Europe and the Middle East it was known roughly how large earth was from Eratosthenes calculation. Did they think the Atlantic ocean was massive and continued until Asia? Or had they speculated on the possibility of other continents existing in the yet unexplored areas of the planet?
Yes actually! From antiquity there was a supposed massive southern continent, which was assumed to be necessary in order to balance out the weight of Europe and Asia in the North. This hypothetical continent was named Terra Australis and lent its name (as you can imagine) to Australia, which was considered to be the most southerly continent after it's discovery. Antarctica was not actually sighted/ officially discovered until the 1820's. Ironically, the hypothetical Terra Australis was sometimes drawn onto maps by early modern European cartographers in more or less the exact area we are used to seeing Antarctica and Australia on Mercator projections today. See here
In addition to /u/theislander1069's comment, I would like to point out the ancient Greeks and Romans occasionally hypothesized the existence of other continents as well, and Pliny the Elder's Natural History makes mention of the belief that those continents are inhabited:
We maintain, that there are men dispersed over every part of the earth, that they stand with their feet turned towards each other, that the vault of the heavens appears alike to all of them, and that they, all of them, appear to tread equally on the middle of the earth. If any one should ask, why those situated opposite to us do not fall, we directly ask in return, whether those on the opposite side do not wonder that we do not fall. But I may make a remark, that will appear plausible even to the most unlearned, that if the earth were of the figure of an unequal globe, like the seed of a pine, still it may be inhabited in every part.
Later in the book (chapter 68) he goes on to describe the temperate zones that humanity could thrive in, being separated by the Sahara, which he presumes circles the earth (where possible).
The middle of the earth, over which is the orbit of the sun, is parched and burned by the flame, and is consumed by being so near the heat. There are only two of the zones which are temperate, those which lie between the torrid and the frigid zones, and these are separated from each other, in consequence of the scorching heat of the heavenly bodies.
Exactly how many continents there might be, and the nature of those continents was obviously under continual debate. I have not had the opportunity to read Ptolemy's Geographia, but he apparently posits the existence of three unknown continents which he named Perioeci, Antoeci, and Antipodes based on the ideas of mathematical symmetry and balance.
