What are some possible effects of the judicial branch being viewed as the weakest? For example, could it lead to judicial overreach? What was the main purpose of the judicial according to Hamilton?
I'll try to respond to all your questions in one answer, I'll start with your third question and then go from there. I will use Publius to refer to Hamilton since both him, Madison and Jay are considered throughout the federalist papers as a single thought/person.
What are some possible effects of the judicial branch being viewed as the weakest? & What was the main purpose of the judicial according to Hamilton?
The reasoning behind Publius' assumption that the judiciary branch of government is weaker than the legislative and executive branch is based on Montesquieu's L'Esprit des lois (The Spirit of Laws), published in 1748 and which heavily influenced 18th century politics. It is still considered as one of the main treatise on political theory. In addition, you can find Montesquieu's influence throughout the Federalist Papers.
As Publius explains in Federalist 78, It is considered the "weakest" for two reasons.
First, because of its functions, as dictated in Article III of the US Constitution, the Supreme Court has no influence on the political rights of the Constitution. The same cannot be said for the two other branches. The judiciary has no influence over the sword (executive) nor purse (legislative) and has therefore, according to Publius neither force nor will, only judgement. (see Federalist 78).
Second, because it can not effectively attack either the legislative branch of government (Congress) or the executive (the Presidency). Everything has to be done in the Constitution to defend the judiciary branch in the best way possible from the other two branches. To do that, it is important that the judicial branch of government be as independent as possible from the legislative and executive branch;
"there is no liberty, if the power of judging be not separated from the legislative and executive powers.''
Again, Montesquieu is quoted in Federalist 78.
However, if you read Federalist 51 (Which explains the checks and balances "Ambition must be made to counteract ambition."), you will see that for the nomination of Supreme Court Justices, there is a carefully worded exception for that rule:
it is evident that each department should have a will of its own; and consequently should be so constituted that the members of each should have as little agency as possible in the appointment of the members of the others. Were this principle rigorously adhered to, it would require that all the appointments for the supreme executive, legislative, and judiciary magistracies should be drawn from the same fountain of authority, the people, through channels having no communication whatever with one another [...] In the constitution of the judiciary department in particular, it might be inexpedient to insist rigorously on the principle: (1) first, because peculiar qualifications being essential in the members, the primary consideration ought to be to select that mode of choice which best secures these qualifications; (2) secondly, because the permanent tenure by which the appointments are held in that department, must soon destroy all sense of dependence on the authority conferring them.
So, if the President and the House of Representatives (The Senate was chosen by the State Leglisatures before the passing of the 17th Amendment in 1913) are to be chosen out of the people, one must make sure that a member of the Supreme Court is effectively the right person for the job, but Publius acknowledges that the population of the United States cannot be the judge of that. So, the process first goes to the President who nominates and then through the Senate who confirms (or not)(1). And in order for Supreme Court Justices to not be dependent on the powers that put them in their seat (Presidency and Senate), their appointments are for life (2).
The argument for independence for the judiciary here is key, because Publius acknowledges that if the judiciary forms a union with either of the two other branches this could form a threat against, liberty, security and justice.
But is Supreme Court really as independent today as the Founding Fathers thought it would?
Could it lead to judicial overreach?
That depends on who you're asking the question to, since no real objective answer can be given to that question. I don't think you'll find a definitive answer given the complexity and evolution of American politics. I'll try to give a few examples, feel free to let me know if that was the type of answer you were looking for. I'll try to be as objective as possible, but note that this is topic that is still debated among historians and political scientists.
As explained in Federalist 51, "Ambition must be made to counteract ambition", but can this also be applied to the judiciary? Not in theory, since neither the ambition of the President nor the Ambition of Congress, can serve as a barrier against the ambition of Supreme Court Justices, since it is assumed that most justices will outlive a presidency or a congressional election/meeting of Congress. In other words, The Supreme and federal courts must exercise a form of self restraint when it comes to the constitution, rather than to be restrained by others. They must be devoted to the constitution and the constitution only.
As Publius argues, the role of the federal judiciary is to protect individual rights from the political intrusion by the other branches of government. But if individual rights need to be defended against political intrusion by the judiciary, then that means that the latter cannot be involved in any political process whatsoever in order to remain impartial. However, it has been argued that today's Supreme Court has become more and more influential in American politics, and this is something that Publius did not intend to happen. I have listed a few examples below.
Over the course of the 1930s, in an era of economic depression and uncertainty in which President Roosevelt put forward his New Deal agenda, the Supreme Court opened the boundaries between state and federal powers in favour of the latter through a number of decisions that were contrary to Publius ideas expressed in Federalist 39 (ex: United States vs. Sprague, upholding of the National Labor Relations Act in 1937,...)
the outcome of Baker vs. Carr in 1962 gave the right to federal courts to review redistricting cases, which consequently gave more political power to urban areas as they were largely under-represented compared to rural areas, especially in the South. However, Anthony Peacock argues that this goes directly against Article I Section 4 of the Constitution which says that only the states or Congress can regulate the time, place and manner of congressional elections.
In The Case Against the Supreme Court by Erwin Chemerinsky, the author argues that in the 2010 case, Hui vs. Castenada, "it would not have cost the court any of its political capital to rule in favour of Castenada" (p.335) and this would not have hurt the court's legitimacy in any way. Yet the court ruled the other way in a 9-0 outcome.
These examples, an many others, show that the Supreme Court can still have the final say in American political debates, which is obviously not what Publius intended. However, there little to no remedy for this issue since impeachment of federal judges can only occur in cases of treason, bribery, and other high crimes and misdemeanour.
In the eyes of Publius, judicial self restraint was the dependent factor for the Supreme Court's judicial review powers, which led to an exception of the Federalist 51 prerogative of one branch's ambition to counter the ambition of another. According to Robert H. Bork, this exception was aimed to find a balance between majority rule and minority rights (i.e. the Madisonian Dilemma) through an impartial and unaccountable institution; the judiciary. That is how it was written in theory by Publius, but as a lot of 20th century example show, this was not always the case.
References
Publius, Federalist 39
Publius, Federalist 78
Publius, Federalist 78
Anthony Peacock, How to Read the Federalist Papers (I highly recommend this book if you want to read more in-depth information on Publius and the Federalist papers. I would also recommend reading it before you dive into the 85 essays)
Erwin Chemerinsky, The Case Against the Supreme Court
Robert Bork, The Tempting of America: The Political Seduction of the Law