Perkin's wikipedia mentions the relationship, however I was unable to find any first hand sources documenting on this matter. What interests me is how this relationship was made known and how she came to accept the relationship.
Perkins's papers are housed in the archives at Columbia University, and there are a number of personal documents, interviews, and correspondence included in the collection. Unfortunately, the documents aren't available online, but based on secondary sources using the archive and interviews with Perkins's only daughter, no direct evidence exists of a sexual relationship between Perkins and Mary Rumsey, or between Perkins and the woman she shared a home with after Rumsey's death, Caroline O'Day. Most of the evidence regarding Perkins and Rumsey's relationships comes from outside sources, such as letters between their acquaintances, and newspaper articles. I should point out that Perkins's daughter, Susanna, has vehemently denied that Perkins had any kind of sexual relationship with Rumsey.
It was no secret in Washington that Perkins lived with Mary Rumsey for twelve years and that they behaved in a manner that reflected traditional gender roles. Their home was the site of gatherings for Washington's political elite, where Rumsey played host. Rumsey attended social events in Washington as Perkins's partner, kept the household for Perkins, and helped to raise Perkins's daughter while Perkins was working. It was also noted that, although Rumsey was a wealthy widow, and still fairly young when her husband died, she never tried to find another male suitor and chose to live with Perkins instead.
It's fair to say that Perkins was queer, but as with a lot of queer history, this is based on educated speculation concerning historically specific gender roles and the way that women were expected to interact with one another. Perkins appears to have had many close female friendships, but none were as close as the one she had with Mary Rumsey. Unfortunately, according to Kirstin Downey - who wrote a great book on Perkins, which you should definitely check out! - Perkins burned most of her personal correspondence.
If we want to locate queer people in history - which is a worthy, but challenging goal - we have to take some degree of liberal interpretation with historical sources. To be clear, this speculation must be educated, and remains a valid method of inquiry for investigating taboos. There doesn't appear to be any direct evidence of Perkins having a sexual relationship with Rumsey, but they behaved as a couple and ran their household as a couple. When Rumsey died, Perkins was the one at her bedside, and Perkins was the organizer and most notable mourner at Rumsey's funeral. Observers noted that Perkins mourned Rumsey in a manner more similar to a spouse than a close friend.
On the flip side, we also have to keep in mind that there were a number of reasons why women may have chosen to live together rather than marrying since - and I know I'm harping on this - the sources simply aren't there. What we know is that Perkins struggled with the dissonance of being a career woman and a wife at the same time, especially as her husband descended into deep mental illness and needed constant care. She was married the whole time she lived with Rumsey, but the living arrangements with Rumsey proved easier for her than remaining at home. In addition, Rumsey facilitated a comfortable life for Perkins in Washington that she wouldn't have been able to afford as she invested money in her husband's mental health care.
As far as how Perkins reconciled her queer relationships as an Episcopalian, sources don't exist on that either. There doesn't appear to be some sort of tell-all document in the archives where Perkins dwells on her sexuality, and she was - according to her acquaintances - a fairly secretive person when it came to emotional matters. That may be one of the reasons she burned her personal correspondence. Historical sources documenting explicit sexuality are, by and large, few and far between when it came to taboo sexualities. There are a few notable exceptions, which are hashed out and thoroughly examined in books like Jack Halberstam's Female Masculinity, if you want to know more about the ways that historians find LGBTQ individuals in history.
SOURCES:
Downey, Kirstin. The Woman Behind the New Deal: The Life of Francis Perkins, FDR's Secretary of Labor and His Moral Conscience. New York: Doubleday, 2009.
Halberstam, Jack. Female Masculinity. Durham: Duke University Press, 1998.
"Notable New Yorkers: Francis Perkins." Columbia University Libraries Oral History Research Office. http://www.columbia.edu/cu/lweb/digital/collections/nny/perkinsf/introduction.html