I'm going to begin by pointing out a common misconception that is implied in your question, which is that there is a uniform (...pun not intended, but I'm leaving it in) group of Jewish people that can be categorized as "Haredi" or "ultra-Orthodox." In reality, the term "ultra-Orthodox" encompasses a number of different groups, including Hassidic Jews and non-Hasidic Jews and the wide range of groups that the categories of "Hasidic" and "non-Hasidic" encompass. These groups have their own philosophies, religious customs, and, yes, way of dressing. To an outsider all ultra-Orthodox men may seem to dress the same, but there is actually more variety than an outsider might assume. There are distinctions between how various different groups dress, and something as seemingly insignificant as, for example, whether your coat buttons from left to right or vice versa or the length of your coat, the kind of hat you wear, or color of your socks and how you wear them, will act as signifiers to those in the know about which group you belong to. 

Which makes sense because one of the roles that fashion plays in our lives-regardless of which fashion rules or trends we adhere to-is defining our identities and membership within a particular community (or as being pointedly not a member of a particular community). For example, if I'm looking at pictures of you from the '90s and you're wearing a loose plaid shirt and Doc Martins, I'm going to get a sense of you and who you hung out with. Ultra-Orthodox fashion essentially plays the same role of allowing wearers to demonstrate their belonging to a particular group, and for those in the know to understand who belongs to which group and who doesn't.

(I also want to point out that what you define as the outfit of Hareidi Jews-namely, the white shirt, frock coat, black hat-are features Ashkenazi ultra-Orthodox men's fashion. Sephardi ultra-Orthodox men had/have their own fashion traditions, but many Sephardic ultra-Orthodox men adopted Ashkenazi ultra-Orthodox men's fashion once they began arriving in Israel. The Ashkenazification of Sephardic religious practice, including fashion, is a really interesting topic but is not one that I feel at all qualified to address. I did want to mention it though, and note that what I am discussing is referring exclusively to Eastern European Jews. Also, your question did not ask about women, which is a whole other kettle of fish, so I'm going to discuss only ultra-Orthodox men.)

Religious Jewish men's fashion has three major influences:

1. Biblical and rabbinic law, some of which you referenced

2. Rules imposed by community leaders

3. Rules imposed by state authorities

Regardless of where they originate, it is safe to say that the purpose of these fashion requirements is distinguishing Jewish people from non-Jews. The motivation, the extent to which the community was willing to adopt the new rules, and whether this was a way of strengthening community ties or something more malicious obviously depended on who was making the rules. During the 16th-century in Poland and areas in Western Europe, Jews were required to wear distinctive hats or other marks that would have made them immediately identifiable as Jews. There were also rules forbidding Jews from wearing certain things, such as wearing clothes or jewelry considered to be too luxurious. These were rules imposed on Jews by Christian authorities for the purposes of highlighting the "otherness" of Jews, keeping them in their place (no showing off), and keeping them separate from Christians.

At the same time, there were rules established by Jewish community leaders that were sometimes incredibly similar. During the 17th century, the Council of the Four Lands (the Jewish governing body) declared that Jews should not dress too ostentatiously; the Council also ruled that Jews should not wear "gentile clothing." So there is a similar desire to distinguish Jewish people from non-Jews, but this time it is coming from within the community for the purposes of preserving the community by maintaining a sense of uniqueness and separateness-as well as preserving peace within the community, so that you don't have some people dressing incredibly ostentatiously while others can't, thereby leading to jealousy and all kinds of bad feelings. If it seems like I'm being more forgiving of the Council's rules than the authorities' rules even though they are very similar in motivation and outcome, that is because I am: a community choosing to adopt dress codes in order to distinguish themselves and preserve their culture is a far different story than having those distinctions imposed on them from the outside.

So now you have this idea that Jewish people dress differently, whether that's because they have chosen to do so or because they have been forced to do so. But, what can you do, when you are a minority interacting with a majority culture you are bound to be influenced by that culture. Perhaps the biggest moment for Jewish men's dress can be traced back to the 16th century, when Jewish fashion started taking on elements of clothing worn by Polish gentry. This influence can be seen to this day in clothing worn by ultra-Orthodox men, which evolved from those styles. These styles were often long and flowing, and corresponded with the weather in Eastern Europe (so if you see a Hasidic man wearing a shtreimel, a fur hat, on a Sabbath or holiday or during a family celebration, that probably evolved from a hat worn by Polish gentry during the winter). I'm just giving a very broad overview, but if you'd like more details about specific items of clothing and how they evolved from the styles worn by 16th- and 17th-century Polish gentry, I highly recommend the YIVO Encyclopedia's article on dress, which is excellent (I'll put a link below). As the Hasidic movement started gaining steam in the 18th century, various rebbes established their own rules and regulations regarding clothing, such that followers of a particular rebbe would be able to identify each other based on their clothing (as I said previously, these distinctions would not necessarily have been noticeable to an untrained eye but they would nonetheless by significant to adherents). 

Re-enter non-Jewish authorities trying to impose different fashions on the Jewish community. But this time, instead of attempting to further alienate and other Jewish people, the goal was to erase their distinguishing characteristics, in an attempt to force them to assimilate. During the first half of the 19th-century, the Russian government attempted to legally force Jewish people to abandon their traditional dress. These attempts met with mixed success in terms of enforcement and compliance, but what it did do is increasingly push more traditional Jews to refuse to give up their distinct manner of dressing. There is a well-known (to religious Jews, at least) tradition that one of the ways in which the Israelites survived slavery as a people is that they did not change their distinct manner of dressing. So the principle of maintaining distinct dress took on religious significance and the force of religious obligation, particularly during a time when the authorities were attempting to undermine the Jewish community via fashion assimilation.

Bottom line: when you see an ultra-Orthodox man and wonder why they are dressed that way? Chances are their clothing has evolved from the styles worn by 16th-century Polish gentry, with elements added by leaders of whatever movement or stream he belongs to, and a sense of religious obligation to wear the specific clothing that he is wearing. Just like if he were wearing skinny jeans, a plaid shirt, and a beanie you would be able to identify him as belonging to a particular subculture, his clothing distinguishes him as being Jewish, and ultra-Orthodox Jew, and for those in the know, an adherent of a particular subgroup within ultra-Orthodox Judaism.

Sources:

https://yivoencyclopedia.org/article.aspx/Dress

A History of Jewish Costume, Alfred Rubens

"Jewish Costume in Sixteenth-Eighteenth Century Poland," Irena Turnau

A Cultural History of Jewish Dress, Eric Silverman