We actually have quite a bit of documentation about this, especially for The Empire Strikes Back, that's been dredged back up due to the lackluster fan response to The Last Jedi (mostly by people trying to justify said response with 'well look at how people responded to ESB, and now it's considered the best movie!').

There are several articles (examples here, here, and here) examining the fan reaction to The Empire Strikes Back and some of the issues some people had with it: namely, how the Vader/father reveal felt contrived, ship wars between the Luke/Leia and Han/Leia shippers (as per usual in fandom) who were mad that Lucas had sort of unexpectedly resolved the love triangle by having her choose Han (remember, the revelation that Luke and Leia were siblings wouldn't happen until three years later), the apparent open-endedness of the movie, hard sci-fi fans complaining about the fact that Star Wars was a space fairy tale/space fantasy instead of hard sci-fi like Star Trek, and (one of the most interesting ones, imo) discussion and criticism due to the scene cuts, which some people felt distorted the sense of time in the movie. You can also read some of the critics' reactions here, and at least one of the NYT critics' reactions was ambivalent at best. The real kicker, in my opinion, is one fan's absolutely furious response to the now-iconic "I know" response:

Somebody should tell Harrison Ford that when a woman passionately tells a man, “I love you,” “I know” is not an an acceptable response. That scene was not funny, it was infuriating. (1980)

By the way, here's an article looking at the fan rumors for what would happen in ROTJ; it's an amusing read.

Reaction to ROTJ is a bit harder to come by. ROTJ made more money in theaters than ESB (ESB had the lowest box office numbers of any of the original trilogy), so we can infer that it was more popular than ESB at the time. However, we do have, thanks to Archive.org's extensive scans of the Starlog zine, some thoughts and reactions from ROTJ (you can check them out in Issue #76 here). While fan reaction was mostly positive, there seem to have been three or four "big" issues with the film: the presence of the Ewoks, Obi-Wan's "five-minute crash course" of exposition where he explains the Anakin-Darth Vader connection, the revelation of Leia as Luke's sister being contrived and out of left field, the deja vu of the plot, and, funnily enough, the fact that there were "too many aliens". The in-depth review of ROTJ later in the same issue is stunningly negative, tbh: from dragging the plot as "nonsensical and disjointed", saying the script was terrible, taking shots at the Luke-Leia sibling revelation, saying that Lucas must have had "creative fatigue" when he made the climax taking out another Death Star, being genuinely irritated with unmasking Darth Vader, and saying that "ROTJ existed entirely to tie up the loose ends of the first two films," it's....not a very flattering review.

The Ewoks, of course, remain a relatively controversial aspect of ROTJ up to the present day, at least amongst older Star Wars fans (my father still dislikes them to a certain extent and calls them "the three-foot-high killer teddy bears from outer space"), and the plot re-run of a Second Death Star has been a standing joke in the fandom for years, but most of the other issues with ROTJ have since either passed the test of time or removed themselves as people re-watched it and the prequels fleshed out the Star Wars world even more.

Overall, both films had a positive response at the time of their release (though ESB had a much more lukewarm reception than ROTJ) and made quite a bit of money at the box office, but there were quite a lot of people whose reception to the movies (and ESB in particular) was ambivalent and even straight up hostile.