Expanded from an earlier answer of mine

This question is asking two things. First, on a more tactical level, there is the question of why did Yamato and Nagato turn northwards at ca. 0755 which pulled the ship away from Taffy 3 and the carriers when the two battleships were relatively undamaged. Second, there is the Kurita's overall withdrawal for the IJN forces at 0920 away from the American escort carriers and transports. Both decisions by Kurita earned him a degree of opprobrium at the time and in the postwar period with one common metaphor employed casting Kurita as Hamlet unable to make a decision until after the die had been cast.

Of the two decisions, the Yamato turn is the more explicable. Lookouts had sighted torpedoes from an earlier torpedo attack and Kurita's battleships faced two choices to "fan the torpedoes", a turn to port (north) or starboard (south). The starboard turn would have kept the fleet into contact with Taffy 3 but posed two disadvantages. Firstly, it would have been steaming into the torpedo line, risking a hit. Second, it meant that both battleships were in serious danger of colliding with the battleship Haruna, which had been steaming in a line parallel to Kurita's division. This particular aspect of the division's decision is one that only very recently become apparent. Older historiography on Samar such as Morrison tended to place Haruna in a different location than more recent work like Robert Lundgren or John Prados which triangulates the battleline more closely using surviving Japanese and American accounts (and note, this still is reasonable conjecture, but still a hypothesis). The officers in charge of the ships' maneuvering instinctively chose the course that minimized damage, with the result that the two most powerful surface units at Samar were now seven miles out of position and its commander lacked even more situational awareness.

This uncertainty shines some light onto Kurita's second turn away from battle. When the US Strategic Bombing Survey (USSBS) interviewed some of the surviving Japanese naval officers, they gave some inkling that they knew after the battle that the opposing naval force was much smaller than they first thought at the start of the engagement. USSBS NO. 170 The Battle off Samar's interrogation of Commander Otani Tonosuke, Operations Officer on the Staff of CinC Second Fleet has a very instructive portion on this very issue and how Japan's lack of air cover contributed to the fog of war:

Q. What type of carriers did you believe they were?

A. We gave that question much consideration, but never fully made up our minds. We found ourselves perplexed by your carriers because they did not correspond to their photographs, and first we thought that they were regular carriers; but after the battle, we decided that they were auxiliary or converted carriers. Also we received word from the tops that there was another formation, and at that time we wondered if we were not confronted by 12 or 13 carriers in all; but this was not ascertained on the bridge.

Q. Was there any attempt to engage in battle with the second group?

A. First, we would encounter the first group, and then take on the second.

Q. What damage did you inflict upon the first group you engaged?

A. One carrier sunk, one light cruiser, one heavy cruiser and one destroyer. There was some confusion between the high gunnery control platform and the bridge. There may have been a repeat report which was understood as two carriers sunk; the bridge concluded that one carrier was sunk. Again from later reports which may have contained duplication, we concluded that we had sunk four carriers, two or three cruisers and two or three destroyers. That was the total result of the day. I now think this is rather accurate, and from a report of search planes at about 1100, we received information that one battleship was severely damaged and dead in the water.

As Commander Otani's interrogation made clear, the IJN's attempt to ascertain the reality of the combat situation was made difficult by the reliance upon visual sightings from surface ships and the constant Allied air attacks on Kurita's fleet. Kurita's USSBS interview likewise suggested that visual surface sightings were inadequate to give the IJN admiral a proper estimate of the situation:

Q. What type of aircraft carriers were the American carriers present? Were they the ESSEX or ENTERPRISE class? Did you recognize them?

A. I don't remember. Starboard bridge structure was all I could tell. There wasn't enough visibility nor adequate reports from the scouting planes.

Although he was not available for a USSBS interview because of his foolhardy Kamikaze mission. Admiral Ugaki Matome's diary entries for the Leyte battles also notes that Allied air attacks and poor Japanese reconnaissance doomed Japan's efforts to throw back the Leyte invasion. Although Ukagi's diary took Kurita to task for confusing orders, he placed a good deal of blame for Japan's defeat on the inability of the Philippine's airbases to provide the surface fleet any form of cover or information. His 24 October entry noted that:

Unless we get enough cooperation from our base air forces, we can do nothing about [concentrated American air attacks], and all of our fighting strength will be reduced to nothing at the end. In such case we should perish by fighting an air battle, hoping it to be a decisive one.

Once the gravity of the defeat off of Samar sank in, Ugaki's 25 October entry pinned responsibility for failure "in some respect to [the operation's] planning, [but] mostly to the extreme inactivity of the base air forces. probably hindered by bad weather." One of the themes of Ugaki's remaining entries for October and November was lambasting Japan's lackluster efforts to produce as many aircraft as possible to turn back the Allied tide.

Although Kurita asserted in his interview that he did not expect air cover from land-based aircraft, Ugaki's diary entry indicates that at least some IJN officers expected an effort to be made by land-based planes. Otani placed a great deal of onus on the Sho operation's failure to poor coordination between Kurita and Japanese air assets:

Q. Where do you think this whole operation broke down? Why did it fail?

A. I feel that from the very beginning that the cooperation between the Task Force (OZAWA) and the Surface Force (KURITA) and the land-based Air Force was bad from the beginning.

Q. What do you feel caused this poor coordination?

A. Coordination between the Surface Force and the (carrier) Task Force was almost impossible due to the restrictions on communication and the need for radio silence; therefore, the plans for cooperation were not carried out. This lack of information from OZAWA was one of the main factors in the failure of the operation, but perhaps the biggest factor was the lack of protection from our land-based air against your (carrier) Task Force. I feel also that the original plan was too complex and inflexible to work properly.

In this context, Japanese commanders focused much less on the small size of Sprague's CVEs and instead more on the number of failures of Japanese planning and execution as the reasons for defeat. The effectiveness of the CVE's air attacks, which Otani observed were some of the most effective air strikes despite their small size, further underscored the disadvantages of surface ships operating in an environment where the enemy held complete air superiority. Combined Fleet CinC Admiral Toyoda Soemu's USSBS interview defended Kurita's actions to retire because of this material disparity:

Q. Under the circumstances as they are now known, in your opinion what that decision of KURITA to turn back a correct one?

A. Looking back on it now, I think that withdrawal was not a mistake. At the time I did not have and Combined Fleet Headquarters did not have information regarding the details of the engagement. Later, when we learned that Admiral HALSEY's Task Force was further south than we thought it was, I believe that Admiral KURITA then would have been within the range of air attack from your Task Force, so that it was not unwise for him to have turned back at that time.

Q. You would not criticize his action now in turning back?

A. I would not criticize.

While some zealots like Ugaki saw defeat as an urgent reminder for Japan to redouble her war efforts to achieve victory (upon hearing of Hiroshima his 7 August entry expressed a wish that Japan should develop an atomic bomb of her own), other IJN officers adapted a more fatalistic attitude towards the war.

Which time? Yamato turned away twice. The first time was in response to torpedo attack, and the second time was at the end of the action.

Up to that final point, Kurita's Center Force had been in combat for 72 hours, and had lost the other most powerful battleship, Musashi, to air attack, and by the end of the Battle off Samar had lost seven of his ten heavy cruisers. Kurita had to swim for his life at the very start of the battle when his flagship Atago was sunk underneath him two days before running into Taffy 3. His men were exhausted.

Importantly, Kurita never realized he was fighting escort carriers and destroyers. His men identified them as US fleet carriers and heavy cruisers, and he believed he'd sunk two of the carriers and two of the cruisers. The ferocity of the defense by the escort did nothing to dissuade him of this, and the volume of air attack put up by the Taffies would have helped confirm his belief. He had not known that Ozawa's Northern Force had succeeded in its decoy attempt and that the Third Fleet was well to his north.

Finally, Kurita claimed to have received a report of US fleet units to his north, and he was attempting to reform his scattered force to engage them. It's likely that this was another intelligence failure and that the air reconnaisance had misidentified his own stragglers for US fleet units. When he couldn't find the new force, he withdrew thinking that he'd not be able to catch a fast carrier force.

Here's a look at Kurita's decision from the US Naval Institute's Naval History magazine: http://www.ww2hc.org/articles/kuritaschoice.pdf