First, let's talk about the formation you see in the movie.

While it's very aesthetically pleasing, it wouldn't hold up well in combat against a major enemy force.

Why not?

The reason why spear walls, and by extension, most infantry formations, work well is density and mass of the unit. When well trained and acting as a cohesive unit, formations of soldiers gain a significant amount of mass and density superior to that of a similar number of opponents that are not in a cohesive unit. Part of this benefit is physical (in that the bodies of the formation are literally packed closer together) but also psychological, as soldiers feel more confident when his flanks and rear are protected, and being able to feel his comrades by his side.

This cohesion of a unit allows a good formation to hold their line against a charge as well as lending strength to their own charges. This indeed also applies to cavalry charges. Much of the threat a spear wall/infantry formation gives to a cavalryman is that a dense formation is less likely to provide a good path to charge through. Indeed, if the formation doesn't falter and loosen up (which is what heavy warhorse riding shock cavalry depend on to conduct a successful frontal charge), it will be absolutely disastrous for the horseman, nevermind the horse.

Stringing out your men like in the Red Cliff 'tortoise formation' is not very wise. It gives up the main advantage of infantry formation vs cavalry.

Moving on, were complex formations in battle a thing in Chinese military history? Yes and no.

Chinese military tacticians, through out the ages, have argued and thought about how to conduct battle efficiently, intelligently, and successfully, much like tacticians anywhere.

Different 'styles' of formation were definitely a thing. Martial writings often refer to a named formations but it definitely isn't like what is shown in Red Cliff.

For one thing, pre-modern communications during a battlefield were notoriously difficult. Various military traditions had various ways of dealing with the chaos of the battelfield and the challenge of maintaining a line of communication from command elements to soldiers in the field.

Various methods ranged from runners, messengers on horseback, trumpets, drums, smoke signals, flags, etc etc

This meant that changing a formation on the fly was incredibly difficult if only because getting the order out was monstrously difficult.

Then, changing a formation on the fly was also hard because it required incredibly high levels of discipline. As a side note, this is why getting a unit to extract itself once already committed to battle was so hard. By extension, this is why it's important to have reserves, because once soldiers were committed to battle, it would be hard to get them out again. Side side note, this is why men trained marching so much. Marching in formation seems easy until you have to do in the heat of battle, with the threat of death, men screaming, and the smell of filth, the dead or dying, and all manner of bodily fluids, assaulting you from all sides.

So how were formations used?

They were set as a general formation before the battle was joined.

Some elite units may have been able to change some formations on the fly but for any but the most elite units with high discipline, it would only confuse and decrease unit cohesion, which in many cases would negate most advantages, if any, of trying to change formation.

And at the end of the day, it isn't as though exotic formations imparted huge advantages. And the more exotic a formation was, the riskier it was to try and pull off, especially if the soldiers in the unit were not elites, ie well disciplined and trained.

/u/AsiaExpert has given a really good answer, so I'll just add on here with some sources that I have come across.

As mentioned, the formation in John Woo's Red Cliff is a work of fiction. In fact it is based on the other work you've mentioned - the Romance of the Three Kingdoms. The formation in question (in both works) is the 八门金锁阵 (literally: eight gates golden lock formation) which was invented by Luo Guanzhong, author of the romance, to add to the mystique of Zhuge Liang in the novel.

Naturally, this formation has no bearing on factual history, much like most military descriptions in Mr. Luo's novel. And the historical Zhuge Liang did not spare much time for thinking up battle formations either (nor was it his job to do so). That said, there were formations being used for military purposes, mostly to ensure that the army was kept in order and attacked in unison, as well as preventing the retreat from dissolving into a rout in case of defeat.

Although there are multiple records of formations used throughout China's history, the ones recorded in Sun Bin's Art of War is probably the most comprehensive list of the formations used from the warring states period onward. Sun Bin details the following 10 formations: [source: Sun Bin Art of War, chapter 17]

1. square formation
2. circle formation
3. spike formation
4. goose formation
5. hook formation
6. loose formation
7. numbered formation
8. fire formation
9. water formation
10. Xuanxiang formation 玄襄阵

The first five are standard formations - spike mainly in attack, circle mainly in defense, square can be used in either attack or defense, goose for crossbow archers or occasionally cavalry, hook for when chariots are used in tandem with the main army.

From 6-10, the situations are more special.

The loose formation is used when the army is small in number, but tries to create an impression of having more soldiers. This is done by maintaining a certain gap between soldiers, with each group carrying more flags or weapons to create a more crowded atmosphere. In such situations, the motive is to intimidate the enemy into standing down, or to make the enemy reconsider a direct attack.

The numbered formation is used when on the defense - the army is split into smaller tight formations that would be easier to maneuver and harder to break down.

The fire and water formations describe how a fire and naval attack should be carried out respectively. The fire attack is to be used against enemy campsites or locations of importance, while the naval attack is to be swift and in accompaniment with the main land army.

Finally, the Xuanxiang formation is to be used to deceive the enemy in such a way that they cannot anticipate where the attack would be coming from. This is the only formation that is detailed to include movement/change - the army is to look haphazard with flags waving about and loud noises, almost to create an appearance of a rout, when actually there is order and proper movement within the ranks. This would be linked to Sun Bin's ancestor's Art of War - "Amid the turmoil and tumult of battle, there may be seeming disorder and yet no real disorder at all; amid confusion and chaos, your array may be without head or tail, yet it will be proof against defeat." [source: the Art of War chapter 5] This would be the closest to the 'unusual formation' as seen in movies, but its purpose is more so to confuse, rather being a complicated formation with purpose at every level.

And on a side note: please take anything from the Romance of the Three Kingdoms with a dose of salt, especially with regards to Zhuge Liang's military achievements. Historically, the 'empty castle stratagem' 空城计 was never applied by Zhuge Liang, though during the three kingdoms period there were records of Sun Jian 孙坚 (father of Sun Quan), Wen Pin 文聘 (of Wei) and even Zhao Yun using it. Likewise the idea of using ships to waste arrows 草船借箭, was evolved from an incident at Ruxu that involved Sun Quan rather than Zhuge Liang.