Is it because the Muslims can justify their religious aggression?
Just because a conquest was done by Muslims doesn't mean it was done for Islam.
For a bunch of medieval European rulers to decide to invade the Levant is a bit weird, you would think they would focus their attention on lands they could hold on to more easily, places in Europe. The crusades don't make sense unless you look at it from a religious POV.
As for the Arabs and Ottomans their goal was the same as every other empire that existed before them. If you're a leader who controls all of Arabia and you want to expand, the most obvious places to strike would be the Sassanid and Byzantine empires. It's not really a religious decision.
Same can be said of the Ottomans, they made a foreign policy decision to expand their empire based on what would benefit them the most, religious motivation was minimal.
Saying that the Arab or Ottoman conquests were 'religious aggression' would be like saying that the scramble for Africa was religious aggression by Christian Europeans against pagan Africans.