Inspired by an ELI5-thread from earlier I would really like to know the origins of the phrase. Hopefully with some sources in order for the actual answer to be presented.
This came up recently in /r/etymology:
https://www.reddit.com/r/etymology/comments/3oiyjx/pardon_my_french/?