Was the paterfamilias always the oldest male relative? It seems that there would have to be some limit there, otherwise your sixth cousin twice removed who you've never met could hold the power of life and death over you and your children. Or was it simply the oldest man in the immediate household? If so, what happened if he was away for an extended period of time? What if a more senior family member lives just down the street?
I'm really trying to understand how this system might apply to a typical American family today, bearing in mind that most American families do not have the same living situation that most Roman families did. Let's say that I'm a man who is married and has children, and we all live together under the same roof. My father is alive and in good health, but lives hundreds of miles away. My older brother is also married with children and lives in the same town as I do, but in his own home a few minutes away. Is my father my paterfamilias as the oldest man, despite living far away, is my brother the paterfamilias as the oldest man locally, or arm I the paterfamilias over the people who live under my roof (and my father and brother over theirs)?
As a legal matter, the paterfamilias was not the "oldest male relative" in a given family, nor was the legal status (typically) dependent on a physical household grouping.
Rather, a paterfamilias was typically a Roman man who had no living lineal antecedents on the paternal line (e.g. his father and paternal grandfather were no longer alive). He was, to use the legal term, sui iuris (under his own law/right), and could exercise the patria potestas (the father's power) over his lineal descendants and their lineal descendants. Depending on the nature of his marriage he may (or may not) have a similar power over his wife (called manus), but this was not necessarily common as manus was not automatically conferred by the more common marriage arrangements.
A man, regardless of age or location, whose father or paternal grandfather was alive was not typically a paterfamilias (unless he had been formally emancipated through one of various legal fictions). The man's legal status was alieni iuris (under the law/right of another), and though he might, as a practical matter, exercise control over his children, he had no legally recognized potestas over them: his father or grandfather held the real authority.
So to use your hypothetical, your father is the paterfamilias. He has potestas over you, your children, your brother, and your brother's children. He also, notionally, owns all of both of your property, although he may have made a limited cession of proprietary interest to you and your brother (called a peculium). It doesn't matter that he lives across the country; in the Republic men often held high office or overseas magistracies who were, nonetheless, still under the potestas of their fathers!
Were your father (omen absit) to die, you and your brother would both become sui iuris and become the patres of your own familiae replete with independent patria potestas. You and your brother would have no authority over each other.
Sources:
Andrew Borkowski's Textbook on Roman Law
Hans Julius Wolff's Roman Law: A Historical Introduction
EDITs: typos and consistency in translations