How did selective breeding work before people really knew about genetics? What did they think caused traits to be passed down from one generation to the next?
So the answer to this is both "yes" and "no" in ways that are typical of the complications about asking about historical understandings of scientific concepts.
In the "yes" category: people had many ideas about how what we would today call heredity worked. At the most basic level there was of course what were "folk" (non-educated) understanding of "blending" heredity. If you had two organisms and they breed, their offspring usually looks like some kind of blend of the two. That is plainly obvious to anyone who has seen children and their parents. On the other hand, it was also clearly more complicated than that: there were, as animal breeders had known since ancient times, occasional "sports" or "flukes," like the proverbial black sheep that is sometimes born to white-haired parents. There were (as will be noted in a moment) different ways to think about these exceptions to the rule. But if you are asking, did people think about heredity prior to 1865, in a sense, yes, they did, though it is a bit more complex than that.
In the "no" category: prior to the late 19th century, people tended not to see the question in terms of "heredity" and certainly not "genetics, a term that, in its modern usage, derives from the idea of a gene or unit character of hereditary information. The framework that was used (in the Western canon anyway, I have no idea about other areas) was one of development. The question that educated people were trying to answer was not "how does hereditary information transmit over generations" but "how does something small and insignificant-looking, like an egg, turn into something large and complex, like a fully-developed organism?" How does an acorn become an oak tree? How does a massive crocodile grow out of an egg no larger than a chicken's? How does something as complicated and wondrous as a human baby grow out of some sperm and a speck of an egg?
There have been, since the Ancient Greeks onward, lots of speculative writing on this point, about the relative contributions of the man and the woman animal, on the role of sperm and menstrual blood and the womb, and so on. There were various theories, like preformationism, which says that eggs and/or sperm contain tiny versions of the final organisms. The 19th century broad new approaches like embryology (comparing embryonic development of different animals) and, through Lamarck and Darwin, evolutionary concerns.
This was also, it should be noted, the context in which Mendel worked as well. He did not see his work as creating a universal theory of heredity — he saw it as being a sub-case of hereditary traits and actually interpreted it as being anti-evolutionary (there is no room in Mendel's original theory for new variation to arise). So the fact that he is typically taken as the "father of genetics" is a little historically wrong — Mendel, as historians of biology have long known, was not a "Mendelian" by the standards of the 20th century at all, and would have found later genetic work very much a different thing than what he was doing. Mendel's 1865 work was ignored not only because it was somewhat obscurely published, but because it wasn't actually an attempt at a master theory of heredity in the first place. As Bowler puts it:
For [Mendel’s] paper to become useful, however, it had to be read in a context very different to that in which it had been written. We can now see that there was no chance whatsoever of anyone in the period immediately following the paper’s publication being in a position to anticipate this alternative context. To extract the kind of insight that the rediscoverers found in the paper, the biologists of the 1860s would have had to realize that the behavior of Mendel’s ‘variable’ hybrids held the key to the whole problem of heredity. No one was in a position to do this because there was no existing framework into which a model of discontinuous character-inheritance could have fitted. Even if it had been better known, Mendel’s paper would have been ignored because the interpretation we read into today would have been unthinkable to the biologists of the time.
The change from thinking about heredity as development to heredity as information came not long after Darwin's work. Darwin himself was firmly in a "developmental" mindset and only somewhat belatedly realized that heredity was absolutely vital to his theory working. Without an understanding of the transmission of traits, it is hard to talk in concrete terms about how minute changes can add up to major ones, or how new "information" is injected into the ecological system. Darwin's own theory of heredity, pangenesis, was a failure of sorts (it was quasi-Lamarckian in many ways, and involved heredity information being stored in the blood), but it stimulated work in his cousin Francis Galton, who reformulated the problem as an essentially informational and statistical one.
Galton and his followers, the biometricians, applied relatively new statistical techniques to analyzing variation between organisms and generations, seeing the issue as a mathematically-driven form of blending inheritance. They were opposed by people who saw inheritance as a more "jumping" fashion (the aforementioned "sports" for example, are not blends at all, but a "jump" between states — black sheep vs. white sheep, with no grays), known as saltationism. Hugo DeVries postulated that "mutations" were at stake, but this by itself didn't give much explanatory power.
It was in this context — saltationists vs. biometricians — that Mendel's work was "rediscovered" in 1900. DeVries and two other scientists (Correns and von Tschermak) had independently, partially re-derived Mendel's statistical laws, in their push for understanding how these "jumping" traits could operate (Correns appears to have fully derived them independently; the other two are more dubious claims). They realized that Mendel had published all of this much more persuasively before, if you ignored all the aspects that weren't quite right (like the anti-evolutionary aspects; Mendel's data, it should also be noted, was suspiciously perfect, and it is probable that he massaged the numbers a bit more than reality would give you), and reinterpreted them as addressing the problems they were interested in. Giving Mendel all the credit was meant to both reinforce the "natural" origin of these laws as well as deny any one of the "rediscoverers" total credit for them (they were not working together, so they were each happy to deny the others the spoils).
If you combine Mendel's theory with mutation theory (traits sometimes change), and put it all into an evolutionary and statistical framework, a whole lot of heredity falls out of it. Even then, the total merging of all of these branches into the modern understanding of heredity and evolution took another few decades, with the modern evolutionary synthesis, which ended up smoothing over and assimilating the differences between the various "camps."
Which is all a long way to say, that even Mendel didn't really have "knowledge of genetics" in the sense meant by the 20th century, and the road to that was a pretty complicated one, framed largely around asking different questions over time. The term "gene," as a hypothetical unit of a hereditary trait, was coined in 1905, explicitly in relation to the re-interpretation of Mendel's work, and the modern use of the term "genetics" followed afterwards. It is interesting to note how theory-laden even these terms are — you are already essentially talking about a information-based concept of heredity, something that did not really exist prior to the late 19th century.
The most readable and interesting book on this topic, which goes over all of the above and then some, is Peter J. Bowler, The Mendelian Revolution: The Emergency of Hereditarian Concepts in Modern Science and Society (Baltimore: Johns Hopkins University Press, 1989). The quote above comes from page 108.