You're right that it sounds unusual, because it's an uncommon formation in English called a postpositive adjective, as opposed to our usual prepositive version where the adjective comes first. (In "prepositive adjective," the word prepositive is, well, prepositive.) Many occurrences of them in English are set phrases that came in from other languages which use postpositive adjectives as their default structure. In English, French is far and away the most common source of this sort of thing.

The terms often ended up borrowed into English usage as a complete unit, and usually kept their original structure even as the terms evolved from "foreign language term we're using in, e.g., the same way modern lawyers toss Latin around" to "this is an English term with an odd-looking construction."

The construction's rare in day-to-day use but shows up a lot more in administrative and legal titles. In addition to Knights Templar, we've got attorneys general, courts martial, heirs apparent, professors emeritus, sergeants major, just about any heraldic terms, etc. (Note which word takes the plural in all of those.) A lot of culinary terms adopted from elsewhere use similar word order for similar reasons. Legalese uses it a lot too, with terms like "malice aforethought" and it even inspires similar things in slangier uses like "murder one."

Essentially, the terms got yoinked into English from other languages ages ago for one reason or another, hung around, and we just organically got used to them being that way. By the time people were making formal attempts to codify 'proper' English grammar once and for all in a way that made any kind of headway with English speakers in general in the eighteenth and nineteenth centuries, all sorts of exceptions and edge cases like that had been baked into the language for hundreds of years.