Both formations are represented in-game exactly as they sound like. Triangle formation has the troops form a triangular wedge while the diamond formation forms a wedge with the troops tapering off towards the end forming a diamond, albeit with the rear angle blunted into a hexagon. While I have heard of the triangle and wedge formation as common and useful formations for shock attacks I have never heard of the "diamond formation" outside some of the latter total war games. If anyone can substantiate the existence of this formation or enlighten me further as to wedge formations in general I would be delighted.
Yes, in ancient Greece it was first used by the Thessalians. The formation (rhombus in Greek) had most of the advantages of the wedge, while also being a lot easier to wheel in. Asclepiodotus has a good description of cavalry tactics that's mostly about the diamond, and also some diagrams (which apparently are very close to the original). The men at each of the four points would have been officers, and the entire formation could change direction by 'facing' left or right (the individual cavalrymen turning in place), then following the new leader.
Source: I originally read about this in 'Soldiers and Ghosts' by J. E. Lendon (quite a good book by the way), which has a description of some of the advantages and some updated diagrams. According to the footnotes, Arrian (who was a soldier rather than just an antiquarian) also talks about the rhombus, but I couldn't find a translation online. If you can read ancient Greek here is the reference, not that I can tell what's going on (aside from the word 'rhombus' anyway).
/u/cnzmur has correctly cited the fact that the 'Rhomboid' was first associated with the Thessalians, who were well known for their large numbers of cavalry and their strict training regime, allowing them to execute some complex manoeuvres. As he said, J.E. Lendon in "Soldiers and Ghosts" discusses this exact matter and provides diagrams, some of which you can see in /u/cnzmur's link. According to Lendon (98-9), the advantages of the rhomboid formation were that it gave the same 'penetrative' power of a wedge/triangle formation while providing greater security to other parts of the formation and allowing greater maneuverability. Lendon also suggests that the rhomboid formation as used by the Thessalians was even 'better' than the wedge formation that the Macedonians had adopted from Thracians and Scythians, and certainly superior to the 'square' formation used by the other, far less proficient Greeks. Lendon cites "ancient tacticians" for these claims, namely Asclepiodotus, Aelian and Arrian, which I think is evidence enough. So this is why Rome: Total War has characterised the 'rhomboid' formation as an "upgrade" to the 'wedge' formation.
The Thessalians were certainly good. The Thessalians were subjugated by Phillip II and after this point served as auxiliaries in the Macedonian army. To be fair, they were only one contingent of well-regarded horsemen, with Boetian, Macedonian and Thracian cavalry also possessing good reputations (Arrian Anabasis 1.7). The Thessalians performed well at Granicus and Gaugemala. However, the Companion Cavalry was by far the most prestigious unit in the Macedonian army, and as I understand Alexander tended to lead them in a wedge formation. The quality of a cavalry unit is as much a product of its leadership, courage and experience as its tactics, and this is the best explanation for the incredible success of Alexander the Great's cavalry as opposed to the heavy cavalry of later Hellenistic monarchs. So while the Thessalians were definitely good cavalry and the rhombus formation was an effective, complex technique that they used, I would probably suggest (a Greek expert could elaborate on this) that the Companion Cavalry and their wedge formation were more effective cavalry, so formation wasn't everything.
OP, you are describing images with words, could you upload a screenshot?
Can I ask a follow up? Why was the triangle / wedge used? It seems like it would flatten out naturally as forces engaged (the people behind the point moving forward), but then the guys at the point of the triangle are in the heat of battle against greater odds until the rest of the force can catch up.
I am severely undereducated in historical strategy, it would seem.
Neither the triangle nor diamond formations where ever used as actual battle formations for either cavalry nor infantry.
The idea is based on a combination of marching formations, parade formations and highly untrustworthy battle accounts.
In reality cavalry was almost always used in wide rectangles, as that brought the greatest amount of horsemen to bear at once. But that would make for a less interesting game.
Sorce: Hans Delbruek, The History of the art of war.