This question needs a big asterisk attached - it's men's formal clothing that tends toward black and white.

The story starts in the late 1770s and 1780s, a period when men and women of fashion began to show their status by, among other things, wearing looser and less formal clothing - the most obvious example we tend to talk about is chemise gowns (often constructed as tubes of white cotton, held in with a sash) in opposition to silk gowns with a fitted bodice and a waistline seam, but this shift affected men's clothing as well. A coat with turn-down collar and revers (together referred to today as "lapels"), previously suitable for wearing while striding around one's country estate, became more fashionable than the earlier coat without lapels. The lapels served a function as country clothing - buttoning a coat over your chest made it warmer on a blustery day, as did turning up a collar over your neck - but were simply an element of fashion at, say, the Tuileries, where the lapel-less coat had been functioning perfectly well. Another, more relevant aspect of this change in attitude toward what was fashionable included a tendency toward darker colors for men.

By the early 1790s, fashionable men could mostly be found wearing dark blue, dark green, grey, brown, and black coats; their waistcoats and breeches might match (the "ditto suit" of matching pieces had previously been fashionable), but going into the 19th century, it became standard for the dark coat to be paired with white or buff breeches/pantaloons and waistcoat. By the 1820s, darker pantaloons and trousers were being worn, and by the middle of the century they were common; between these periods was a time of colorful waistcoats, but by the 1860s-1870s, they were calming again into the matching waistcoat of the three-piece suit or simply buff.

Stark black ended up remaining in use for evening dress for a couple of reasons. For one thing, as you can see from the above, it had achieved a "traditional" status by the mid-19th century by virtue of being what used to be exceptionally fashionable. For another, evening dress, as formalwear accessible only to the affluent, most fully represented and conformed to the mainstream 19th (and 20th) century philosophies regarding men's and women's clothing: that the color, fuss, and frivolity of fashion was reserved for women, while men should be "above" fashion and remain in soberer attire.

Black does also have a long history of being a prestige color in the west. While of course it has also been used as the main color of mourning, it had been quite expensive to procure cloth of a true, deep, and stable black: it required multiple dye baths (and not the second or third uses of a bath that created lighter versions of the dye color, either) which drove up the cost. Meanwhile, the ability to have one's personal body linen - men's shirts, women's shifts/smocks/chemises - regularly cleaned and kept a spotless white was also a status symbol. Body linens would show to some extent, in different ways depending on the era: for men in the 19th century, this included the collar, the shirtfront, and sometimes the cuffs. The clean, starched, white shirtfront was especially prominent for evening dress, which included a waistcoat with a shawl collar and a very deep neckline.