From a quick bit of research, it appears it originates from the literal use of French in a conversation, and then the user of said language would apologise as not many people would understand it.

Example from a Ladies Magazine from 1830:

Bless me, how fat you are grown! - absolutely as round as a ball: - you will soon be as embonpoint (excuse my French) as your poor dear father, the major.

"Embonpoint" is French for 'plumpness'; state of being well-nourished.