I can't answer your first question directly at the moment, but I can answer the second, and I think that doing so can illuminate at least part of the answer to the first. Regarding the question why the Soviets fell behind after seeming to be so far ahead of the US, there are a few key factors. The Soviet economy just couldn't compete with the American economy in long-term investment in space research, for one; for another, the American technological effort was focused on the moon from much earlier on than the Soviets were; and the organization of the Soviet space program was not actually very well suited to what they were trying to do.

 

Of course, from the US perspective, if you ignore the information that was only available to the Soviets at the time, it does look a bit strange that the USSR would have been doing so well and then lose all of its momentum. However, it's a mistake not only to think that the Soviet Union suddenly lost its superiority around 1965-66, but also to think that it was ever really this insurmountable juggernaut of space technology. The USSR made a lot of impressive progress in its space program from a less economically powerful starting point than the US, but in part the perception of Soviet successes was played up by American politicians because presenting the Communists as a powerful enemy and "being tough on Communism" was a way to win over the electorate, and in part cultural memory has played it up as well, because everyone likes an underdog narrative. Calling the post-WWII US an underdog is, well, not something I personally subscribe to, but there you have it. The point, though, is that the processes that eventually bogged down the Soviet Union's lunar effort were already at play from the beginning, as were the factors that allowed for the United States' successes; there was no sudden switch of preeminence in the mid-sixties. And although it is undeniable that landing on the moon took some incredibly skilled engineering, that was by no means the only reason for course of events that transpired. For both countries, the developments of the space race can be seen as part of one long process.

 

That said, though, as much as I don't like to think of the matter as being easily divisible into two phases, one of Soviet success and the other of American, in part the reason that events developed as they did is that, as I mentioned, the United States started to plan for a moon mission earlier, and the binary between orbital missions and lunar missions is an important one. Part of the reason for that is that American leaders realized in the late 1950s that the US probably would not beat the Soviets to a lot of the achievements to which they, in the event, did not beat the Soviets. As a result, the US emphasized the moon as the ultimate goal of the space race because it knew that it was a goal it had at least a chance of achieving first, which in part answers your first question. We say now that the US won the space race because the people in charge of the effort got to decide, at least in the American public consciousness, what counted as winning. And indeed, in Russia, the lunar effort is seen as a much smaller part of the larger period, and certainly not as the single qualifier for victory.

 

But to return to the earlier point, the US began to conceptualize the moon as the ultimate goal around 1960-61, and geared much of its research towards that objective from that point on. The Mercury, Gemini, and Apollo programs were envisioned as different stages of development towards the eventual lunar landing, and as a result, not only were the data that the US found and the lessons that it learned more related to its lunar ambitions, but also, development on the eventual capsules and rockets that would take Americans to the moon started as early as 1961. All three programs ran simultaneously, until the earlier two were considered successful.

 

In the Soviet program, on the other hand, plans for a serious lunar rocket did not get underway until 1964. Although Soviet propaganda accented the dream of a peaceful socialist utopia in space, the leadership of the USSR knew that the USA had many more nuclear missiles than they did, and they strove to achieve parity in the number of rockets they could field. As a result, the USSR primarily financed the construction of nuclear missiles, which were indeed very similar in design to their low-earth-orbit lifters, but quite dissimilar to the kind of rockets needed to land on the moon. Thus, that financial focus on low Earth orbit helped the Soviet space program in its earlier stages, during the race for the first satellite, the first man in space, and various other achievements that could be done in low Earth orbit, but it actively interfered with the lunar program. Even within the lunar program, sometimes engineers were forced to work on both nuclear missiles and orbital rockets simultaneously.

 

Because of this concentration on the nuclear competition with the US, questions about a manned lunar landing didn't really come to the fore until the Soviets had already achieved many of their well-known successes in low Earth orbit. That was where it lost the advantage, since it wasn't ready for a protracted competition, in which the ability to maintain momentum became a large factor. As an example both of just how different the two kinds of missions were in terms of technology and of how the Soviets were unable to switch from one to the other, the N-1 launch vehicle, which was the eventual result of the Soviet lunar program, weighed about nine times more than the earlier R-7 Soyuz launch vehicle, but there wasn't enough time or money to develop a new rocket engine for it, so the first stage of the N-1 used 30 NK-15 engines arranged in two rings. In comparison, the first stage of the American lunar launch vehicle, the Saturn V, had just five F-1 engines, and each F-1 was about five times more powerful than the Soviet equivalent. In general, the layout of engines on the N-1 was quite inefficient compared to the American solution, and it raised the chances of a problem or even a failure. But under the Soviet organizational structure, this inefficient layout became necessary, and these kinds of cost-cutting and time-saving measures often couldn't be avoided.

 

And that leads me to discuss another hindering factor, which is that from the perspective of organization and logistics, the US was paradoxically more willing to centralize its efforts in the space race. Of course, we in the US love to talk about how the advantage of capitalism over communism was the fact that a free market leads to competition, which then in turn leads to production of higher-quality goods, and about how communism is destined to fail because it leads to complacency and stagnation. Of course, let's not get into the weeds of that debate, but no matter what you think of the matter, it's interesting to note that the United States was the country that gathered all of its efforts under a single logistical umbrella, as it were, namely NASA. Various aerospace companies did compete for federal contracts, of course, but after a company received a contract, that was more or less that; the other companies would withdraw their bids, and within that company there would be a relatively constructive effort to find a single best design, not to compare multiple possibilities. The United States actually behaved more collectively than the USSR in this regard at least.

 

In the USSR, on the other hand, efforts were very decentralized and poorly organized. After the head Soviet designer, Sergei Korolev, died in a heart operation in 1966, various teams of designers began to fight for their own particular constructions, and these battles wasted a lot of valuable time and prevented the maximum exploitation of the talents of Soviet engineers. The USSR's lunar program was just generally held up by various problems, such as Korolev's death, or the failures of four expensive N-1 rockets, with each of which the USSR lost a lot of irreplaceable money and time. But Korolev's death really only intensified already existing organizational problems, and the N-1 only ended up failing each time because the engineers who were working on it were forced to hurry and make mistakes because of the structural disadvantages they were fighting. The difference in economic situations and in organization of the respective space programs were the main reasons that the USSR was unable to beat the United States to the moon, despite their earlier successes.

 

Soviet industry, despite its achievements, and soviet engineers, despite their intelligence, simply couldn't switch over from low Earth orbit design to lunar design quickly enough. And although the USA seemed behind the USSR, in fact, the research and the efforts of American designers were much better and more equally distributed than the approach their Soviet counterparts took, which allowed them to achieve not just the early goals of the space race, but also what they defined for themselves as the later goals.

 

So, why did the USSR achieve so much in the first years of the space race, and yet still lose in the race for the moon? Ultimately, I would say that the most important factor was that the American economy simply was larger and more able to take on the strain of the project than the Soviet economy. That didn't predetermine the result, but it meant that the USSR was facing an uphill battle. The Soviets then proceeded to make the situation worse for themselves by starting to think about a lunar landing much later than the Americans, and they were unable to refocus their industrial and design efforts until it was too late, or even to efficiently organize their lunar program at all.

 

Edit is for formatting, because I don't get Reddit formatting.