"Prefecture" seems an odd and obscure choice of English word to translate the Japanese concept. Why not "district," "county," or "province?" Those are much more commonly used in the English language for similar administrative sub-divisions in other countries.
The Portuguese arrived in Japan first, and they used their word prefeitura, and when the English speakers got there they made the perennial mistake of translating to the cognate 'prefecture' rather than a word closer in actual meaning.
The Portuguese also started using the Indian term 'mandarin' for Chinese government officials, and now half the world calls their language by a Sanskrit word.