had their own secular customs and traditions

I'm less familiar with Byzantine culture but I would extremely hesitant to use "secular" in a medieval context.

jizya would have to be paid

It's actually interesting, it's taken for granted on the basis of later practice and sources that "jizya" sort of means "discriminatory tax on non-Muslims" but by all accounts I'm familiar with it was modeled in the early period on the poll taxes already in place, and its imposition in practice was so varied, including it being levied on converts to Islam, that I would make the case that "jizya [being] paid" would not have struck anyone as being much of a difference from prior practice. The later assertion that non-Muslim subjects during the conquest were offered the tripartite choice to convert, pay the jizya, or die is belied by the fact that A: in the earliest period after the conquest it was very difficult for non-Arabs to convert, B: we know of settlements between the Arabs and conquered peoples where they came to non-jizya arrangements, C: we don't have any examples of people who resisted (such as Alexandria) being put to the sword.

Anyways a bit of an aside, which I'll return to.

did the Caliphate itself develop its own civilization

I'm not sure I've come across attributions of the Ummayad Caliphate being, in and of itself, a civilization, but did they make contributions to Islamic civilization generally? Absolutely. This is tied into one of the more contentious questions in Islamic history, which is "What was the nature of the early Islamic State? Was it Islamic? Was it a State? When did it become so?"

While there is, as yet, no definitive answer for early a state of some kind might have formed, the most concrete answer for when an Islamic State came into being is during the reign Umayyad caliph Abd al-Malik. It's during Abd al-Malik's reign that the imperial coinage transitions from a Perso-Byzantine style representation of the ruler, to a recognizably Islamic style absent of depictions. These are the first coins that mention "Muhammad" and include the Islamic formulas like shahada: "There is no god but God and Muhammad is His messenger."

Abd al-Malik also builds the Dome of the Rock in Jerusalem, which is a fascinating structure. It's precise intent remains somewhat obscure but it's one of the earliest examples of imperial Islamic religious architecture placed prominently and intentionally in one of the holiest places in the Abrahamic faiths, the Temple Mount, and made all the more provocative by its selective inclusion of propagandistic quotations from the Qur'an that are most explicitly anti-Trinitarian and aimed at appealing to Christians.

As I mentioned, where conversion of non-Arabs to Islam in the early period seems to be rare, or at least isn't well-attested to in surviving sources, there is a flood of such conversions from the reign of Abd al-Malik onwards. Where before coins take after Perso-Byzantine models, after him, the Islamic formulation prevails. Where he builds some of the first monumental Islamic structures, after him there are many such constructions.

So when you say that people might notice "some mosques in the area" that's not really something people would have noticed until the Umayyads, who made a point in cities like Jerusalem, Damascus, and Aleppo of building enormous religious structures in non-Muslim spaces.

To return to the jizya, it's in the reign of the Umayyad's, specifically under Umar II, Abd al-Malik's nephew, that we start to see discriminatory policies towards non-Muslims. Prior to this, in Fred Donner's conception, there was less of a religious distinction among the "believers" as they termed themselves and other people of the book (with whom we know early Muslims prayed in the same churches) and that it was therefore part of a policy starting with Abd al-Malik in which "the elite seems to have been engaged in rethinking their own identity as Believers and "drawing the line" between them- selves and those who embraced any hint of trinitarian doctrines."

Robert Hoyland ascribes different motivations and sources for this policy shift, noting:

The stimulus for this seems to have been the ignominious failure of the Arab siege of Constantinople in 717–18 and the huge loss of Arab life. This defeat intensified hostility toward Byzantium, and by association toward Christians, and it also accelerated the professionaliza- tion of the army. Many Arab Muslims relinquished their military role and became civilians, but they did not want to rub shoulders on an equal foot- ing with the non-Muslim conquered peoples. Accordingly, restrictions were placed on the latter to keep them in their subject position. The raw material for these restrictions came mostly from Byzantine curbs on Jews (not build- ing new synagogues, not giving testimony against Christians, not defaming Christianity, etc.) and Sasanian Persian regulations for distinguishing between nobles and commoners (not wearing the same headgear, overcoats, belts, shoes, and hairstyles of the superior group, etc.).

Interestingly, it's only at this period that legal formulas and conceptions like "convert, jizya or die" start to emerge. Hoyland again:

Gradually there evolved an extensive body of legal rulings governing what non-Muslims could and could not do and how they should behave toward Muslims. Jews and Christians and other non-Muslims became a subordinate class, and yet were integrated within the Muslim legal system and granted protection.

Even so, in practice, fiscal reality meant that taxes were far more uniform than some of the later religious legal accounts might suggest. Hoyland:

there was now a uniform land tax levied on Muslims and non-Muslims alike. The third reform was to provide a Muslim counterpart to the poll tax, which had come to be seen as a specifically non-Muslim tax; the solution was to make almsgiving for Muslims compulsory, collected just like a tax. This policy was probably introduced not long before 730...

By the second half of the eighth century, the island of privilege that the early Arab conquest society had been no longer existed. An average Muslim very likely paid less tax than an average non-Muslim, but it varied according to profession and status, and of course the reality of tax collection was very much more complex than the simple and elegant theories of the lawyers."

Ultimately these issues: "Who was a Muslim?" "What was the status of converts?" "What was the status of Arabs?" Played a key role in the downfall of the Umayyads, just as it was the defining feature of their policies from Abd al-Malik onwards. The Abbasid revolt was in large part led and supported by converts and the descendants of converts like the general Abu Muslim. While attempts to describe the Abbasid revolution as proto-Iranian nationalism go too far, it nonetheless played a role in making the Islamic empire and Islam less intrinsically Arab. This in turn, culturally, ends up contributing deeply to the culture of the Abbasid period, which was greatly enriched by non-Arabs, and particularly Persians.

One last example of Umayyad culture that I'd simply be remiss to pass over was the "Desert Castles" that they built, most of which are in Jordan and Syria. Their purposes are somewhat obscure, but as cultural productions they're extraordinarily beautiful, with Roman-style frescoes and richly detailed masonry work. If you're ever in Berlin a trip to the Pergamon Museum which houses the Mshatta Facade from one of these castles along with a number of these frescoes is well worth it.

Source wise:

Most of the quotes are from Fred Donner's Muhammad and the Believers or Robert Hoyland's In God's Path.

I would also take a look at some of their earlier articles on these questions as being worth while, such as Hoyland's New documentary texts and the early Islamic state and Donner's (much older) The Formation of the Islamic State both of which go into a good bit of the historiography of these questions.

Edit: oh and for a good quick overview for non-specialists on the coinage issues I mentioned, there's a quick episode of the BBC's History of the World in 100 Objects transcript HERE that does a fine job.