Oswald did not pass out anti-Castro leaflets, quite the opposite. The leaflets he passed out in New Orleans were a part of the Fair Play for Cuba Committee (FPCC), a pro-Castro group that opposed the American government's hostility to the Castro regime in Cuba. The FPCC itself was something of an outlier within the American left. While there was some overlap in its members with the old guard of the US Communist Party (CPUSA), the FPCC also drew in various other disaffected leftists that were often alienated from the CPUSA and its pro-Moscow bent. Castro and other third-world Communist leaders like Ho Chi Minh or Mao struck a more romantic and dynamic pose than the old men in Moscow. A lot of the CPUSA's leaders like Gus Hall or Eugene Dennis were veterans of the labor fights during the Depression and seemed to be out of step with the younger generations of the American left. Adding to this mix were old-line Trotskyites and Schachtmanites who remained committed to Leninism, but felt that the Soviet dictatorship under Stalin and his successors had distorted the Revolution. Cuba within this specific context and milieu became one area of common agreement for a good many members of the American left, even if the details of their specific ideology differed. In the end, it is helpful not to conceptualize American politics in this period (or any period) in terms of binaries like communist/anti-communist or even left/right, as the politics of this period could be quite flexible.

As to where Oswald fit in this picture, that's an incredibly difficult thing to parse out. Conspiracy theorists have made it difficult to gauge just where he stood and why he did things like defect the USSR and return to the US. Outside of conspiracy circles, most historians tend to view Oswald as someone who was very much politically adrift and searching for some kind of cause to latch onto. When his experience with the Soviet Union did not pan out, he then gravitated towards Cuba. Peter Savodnik's book The Interloper: Lee Harvey Oswald Inside the Soviet Union provides a very good window into Oswald's actions. Savodnik talked about his book for the New Books in History podcast and it is well worth listening.