There was neither strong unity, nor strong discord, but rather societies that were largely separated, or united on pragmatic (rather than ideological) unity. First, the obvious disclaimer that "the Ottoman Empire" covers not only a huge amount of time (1299-1922) but a huge amount of space (Vienna to the Atlantic to the Indian Ocean to Southern Russia).

First, as is famously cited, the Ottoman Empire was arranged around a system of millets (religious communities). As this system reached a strong development in the 15th century (especially after the Conquest of Istanbul), there were four main millets: the Muslims, the Greek Orthodox, and the Armenian Apostolic. Christians were not organized into a new single millet, but rather pre-existing ecclesiastical hierarchies were recognized and as new territories were brought in, they were more or less included in the millet system. Critics, especially Braude (1982) and Braude and Lewis (1982), have pointed out that many parts of this system were formalized surprisingly late. For example, while two of the millets were formally recognized in the 15th century, but one of them (I am fairly sure the Jewish millet) was only recognized on a more ad hoc and personal basis. However, as Karen Barkey and others have pointed out, just because it wasn't formalized in written law, doesn't mean it wasn't part of a coherent (though flexible) system. We can understand millets as one of the guiding organizational principles of the Ottoman Empire, even as we recognize all sorts of practical cases where people violated these principles. For instance, Muslim disputes and disputes between people of two different millets were supposed to be handled by Muslim courts, while disputes between people of the same millet were supposed to be conducted with their own courts. However, we have many cases of, say, two Greeks or two Jews going to the local Muslim kadi in hopes of getting a better ruling. This doesn't mean that there wasn't a millet system, but rather shows that the system wasn't as rigid as it was once made out to be. As another example, Catholics and Syriacs and Druze might not have their own representations in Istanbul, but were largely allowed to "do their own thing" in their own regions.

What we don't see, however, is a lot of ideological clashes. Compared to European Wars of religion, in the day to day, year to year operations of Ottoman Empire we generally don't see Muslim-Christian or Muslim-Jewish violence until the 19th century. What we do see, after the rise of the Safavids around 1501, is (Sunni) Muslim on (non-Sunni) Muslim violence, especially towards Anatolian "Kizilbash". Again, we see a flexible tolerance in other places, such as Ottoman Lebanon where we see Twelver Shi'a and Druze communities thrive at various times. The conflict between Muslims and Christians mainly dates to the late 19th century, with a turning point perhaps being the "Batak Massacre" of 1876, which really galvanized the press in Christian Europe. Nationalism had been creeping in to the Ottoman Empire throughout the 19th century. Greece and Serbia broke away at the very start of the century but, for instance, Peloponnesian Greece's war of independence didn't really lead to violent conflict against the Greeks of Istanbul (though, over the course of the century, it was one of the things that lead to increasing suspicion against them). The issues around nationalism abound, but by the end of the 19th century, we start to see patterns of religious communal violence, especially picking up with the Hamidian Massacres. The exact causes of these are complex, related both to economic changes, legal, international, and ideological reasons. They soon became a largely self-perpetuating cycle (-->Sasun resitance-->massacres-->Ottoman Bank Takeover-->massacres-->) identities become increasingly politicized. This cycle only ends with the creation of mono-ethnic states through the killing and expulsion of the Anatolians Christians (Armenian, Greek, and Syriac) and the ethnic cleansing of Muslims from the Caucasus and the Balkans (though in does continue in more limited forms like the Istanbul Pogrom of 1955). Muslim-Jewish violence was also fairly limited, especially after the Spanish and Portuguese Jews started arriving after their expulsions 1492 and 1496, also until the 19th century. Antisemitism in the form of pogroms and blood libel only emerges in the 19th century and is clearly an importation from Christian Europe that takes on a second life in the Ottoman Empire. Antisemitism to this day in much of the Muslim World has a clear basis in earlier forms of European antisemitism (from blood libels to the Protocols of the Elders of Zion).

People lived, in urban spaces especially, in largely separate communities. Still, there were lots of connections between these communities. Muslims secretly (and not so secretly) went to Greek tavernas/meyhanes. Elite networks around the Sultan (again as documented by Karen Barkey) were quite heterogenous in terms of ethnicity/religion (with recent converts to Islam often acting as important bridging nodes), especially at the height of the Ottoman Empire. Many of the great mosques were often made with Greek and Armenian craftsmen, the Sultans diplomats ("dragomans") were often Greeks, the personal physicians often Jews. Religious minorities in the Ottoman Empire were classic "middle man minorities" dominated urban business (especially after European powers became more involved with the Empire) which was one of the things that helped fuel later violence, and Muslims tended to dominate agriculture (though not universally; for example, Christians were important in parts of the Levant and the Balkans). On a local level, there's ample evidence of Christians and Muslims mixing socially together in rural places (especially mountainous ones) similar, perhaps, to the mixing of Jews and Christians in rural and semi-rural Poland, but Ottoman records are often particularly bad for this sort of "social history" (I have friends who are trying to recreate it from missionary sources, but this work is relatively new and as far as I know mostly unpublished). Again, we see nationalism and ethnic conflict increasing in the 19th century, especially as the Ottomans exchange the indirect rule of earlier times (when regions were essentially run by local notables) with a more direct rule from Istanbul (this is in line with Michael Hechter's theory that the switch from indirect rule to direct rule globally was the main cause of the emergence of modern nationalism).

Mixed in with all of this, were obviously tremendous variations across space within those ethno-religious communities--Jew from Istanbul, Greece, Baghdad, and Yemen might have very different styles of dress and even different home languages and religious customs: Istanbul would be Ladino and "Spanish and Portuguese", Greece might be linguistically Ladino or Yevanic but likely religiously Romaniote, Baghdad would be Judeo-Arabic and Mizrahi which was heavily influenced by but also separate from the Spanish and Portuguese Sephardic tradition, and Yemen would be a different type of Judeo-Arabic and Temani which was largely isolated from the rest of the Jewish world. Similar local differences would play out in the other millets.

So, in short, what we see is neither long patterns of "unity" nor long patterns of "conflict" (at least, before the 19th century), but largely separately organized ethno-religious communities (with large internal diversity) that would cooperate pragmatically but also had largely separate institutions, including not just religion, but courts, schools, languages, place of residence, but in places even things architecture (if you see a cumba window in Istanbul, you know it used to be a minority-dominated neighborhood) and dress (Wikipedia has some great pictures from the late Ottoman Empire).