In iranian mythology "rostam" the aryan champion fights "demons" (دیو) in current province of "mazandaran" so that aryans can live in peace i have been told that these "demons" are the people who were already living in parts of iran before aryans came and conquered them. Any idea on who they were ? what cultures did they have ? anything besides the Elamites
Well, first, I'm going to spend some time addressing some of the misconceptions in that particular theory, but I'll get to your actual question by the end.
One important thing to note about the Shahnameh's story of Rostam travelling to Mazandaran is that it is not the modern province of Mazandaran. When the stories that Fedrowsi eventually compiled in the Shahnameh were developing, that area on the south shore of the Caspian was still called Tabaristan (and before that it was Hyrkania or Varkana) and "Mazanderan" was probably first applied to that area around the same time that the Shahnameh was written. Ideas of where the mythical Mazanderan was located range from Egypt to India, basically suggesting a distant, poorly understood, semi-legendary land, similar to how the Late-Period Egyptians thought of Punt.
Likewise, I should point out that according to the same legends in the Shahnameh and related epics, Rostam was already in Iran, or at leas the area considered Iran at the time it was written down. Rostam was associated with Zabulistan and Sistan, in what is now southern Afghanistan and eastern Iran. In the Shahnameh, Central Asia (where the Indo-Iranians/Aryans migrated from originally) is called Turan.
Both Mazandaran and Turan originate in the Avesta, the collection of Zoroastrian holy scripture and prayers. Turan is used broadly to designate enemies of Zoroaster's people and Mazandaran is just briefly referenced. The best guess for the inspiration of this mythical Mazandaran is probably somewhere in ancient Inida (plausibly in the modern borders of Pakistan). The Vedic parts of Hinduism share roots with Zoroastrianism and Iranian religion, but feature many gods that are derrided as daiva by the Avesta. This could be the ultimate explanation for the evil magicians and divs/demons featured in the stories of Rostam.
The first point I made is important to answer your original question too. Modern Mazanderan does not make sense given the context of the story, but it is almost certainly one of the first areas of modern Iran encountered by at least one group of Iranians. Based on the linguistics of ancient Eastern Iranian and Western Iranian languages, it's probable that different groups were encountering different parts of pre-Iranian Iran (if that makes sense) at different times.
The eastern wave or Iranians, that moved into what is now Afghanistan and Sistan, would have encountered cultures we know next to nothing about. The archaeology of eastern Iran is very sparsely documented, and the prominent sites dramatically pre-date the events we're discussing. When Iranian-language-speakers were first entering that region around 1000-800 BCE, we really don't know who was there already. That makes it basically impossible to discuss who would have been encountered by a theoretical Rostam.
But if we look to the west, we have a much better idea because those peoples were documented and referenced in their interactions with Assyria, Babylon, and Elam. You've already identified the Elamites in the southwest, and we've ruled out the east, so we're basically focusing on the people who lived in the northwestern Zagros mountains. Of course, we don't know a ton about these peoples either, simply because we're still reliant on other culture's for our information (namely the Assyrians).
The early Iranians probably found all sorts of smaller tribes that outside sources lumped into broader categories, and of course because we're relying on the Assyrians for our information, we really only know what was happening at the very western edge of Iranian expansion, spearheaded by the Medes. By the time any sort of documentation discusses events further east, everyone is already regarded as Median. There are four big, historically well known groups that were in the region before the Medes arrived, and conveniently they each represent a different outcome for those pre-Iranian groups.
First up, and the least well documented, is Lullubi. Lullubi was a small, but persistent culture in the Zagros. They first appeared in ancient Sumerian literature and persisted on for thousands of years. They never appear to have coalesced into a powerful kingdom of any sort, and mostly appear in records as the targets for raiding Mesopotamian armies. Culturally, they were probably a combination of pastoralist herders and merchants who gained some wealth from their position on the east-west trade routes bringing tin and lapis lazuli from modern Afghanistan. By the Assyrian period, they morphed into a kingdom called Zamua, which was urbanized and had fortified cities to protect the wealth of those trade routes. But by the late 7th century BCE, Zamua and the Lullubi seemingly vanish to be replaced by the Medes.
To the south and west of Lake Urmia, Zamua bordered a kingdom called Mannea. The Mannaeans were a more cosmopolitan group that resulted from a few centuries of migration at the end of the Bronze Age. Most of the Mannaeans, or at least their rulers, were descended from Hurrians who originated in northern Syria. The Hurrians mixed in with some Kassites (below), and probably other native groups. The Mannaeans are especially interesting because the Assyrians initially listed Mannea as a separate entity from the Medes, but over time, came to include it as a Median territory. This suggests that the Mannaeans were gradually Iranized, but retained some identifiable political separation from other Median groups.
The Kassites were another pre-Iranian group that occupied the same region. However, thy are better known for the time they conquered Babylon in the 16th century BCE (700 years before the time we're talking about). The Kassites who remained in Iran would not have adopted Mesopotamian culture in the same way their cousins in Babylon did, and as a result we don't know much about them. Their language was not Indo-European, but may have been related to the Hurro-Urartic family (and thus related to the Hurrian language used in Mannea). However, some of their kings had Indo-European names, possibly suggesting that the Kassite ruling class developed out of the same earlier Indo-Iranian migration that produced the Mittani kingdom in Syria.
The Kassites actually survived as a distinct group, not only through the Iranian migration, but through the Median and the Achaemenid Persian Empires and on into the Seleucid period. Not only that, but Greco-Roman sources describe them as one of the tribes that was never fully subjugated by the Persians and would demand bribes or rob caravans in their territory, and seemed to develop a practice of offering mercenary services. Ultimately, they seem to have been absorbed into surrounding (Iranian) cultures sometime in the Hellenistic period, long after all their more ancient neighbors.
Similar to the Kassites, were the Gutians. This was another persistent culture, like the Lullubi, and like the Kassites, the Gutians made their name known in history for conquering Mesopotamia. In their case they brought down the ailing Akkadian Empire around 2100 BCE. That dynasty collapsed by 2050 BCE, and the Gutians never appear as prominent players in regional affairs again, but are referenced repeatedly for the following 1500 years. The Gutians appear for the last time when a Gutian general named Gubaru (an Iranian name) is identified as the general leadin the Persian army of Cyrus the Great against Babylon in 539 BCE. Unlike the Kassites, it seems the Gutians were already Iranizing at that point, and joining the hierarchy of the Median and Persian kingdoms probably accelerated that process.