From what I understand even non-equals used the term ''my lady'' to refer to women of a certain stature. Why was that considered appropriate coming from lesser men (specifically the ''my'' part)?
So, I tried digging up research on the modern usage of "my lady" and nothing really satisfactorily explained it neatly. Basically, it's an honorific, which is why it's not an insult to be addressed like that. "My lord" is the masculine variation, and it is used when showing deference to someone of higher rank. Just like saying "my boss" doesn't really mean you own your boss, but that you work for your boss. The term appears to have been held over into modern usage to show respect for an adult woman, regardless of rank. Again, research is not turning up anything satisfying as to when this happened, but my hunch is that it occurred around the time of the Industrial Revolution. Modernly, it shows up as a form of address by servants to their mistress, again independent of actual rank, which makes me think it was retained when the split between having wealth and having a title occurred (one not necessarily precluding the other after about the mid-19th century).
It's been revived in recent years by 4chan and Reddit as a sarcastic reference to a certain sub-group of young men who are trying, clumsily, to show courtesy to a woman by acting in a "chivalric" manner. Before then, it would be pretty rare to hear anyone using "my lady" or "m'lady" unless they're actually addressing a woman of rank.