Cotton is cultivated since a very long time in the Indian sub continent, however it has very mild success in China until the dual roller gin until 12-14 the century, do we know in which material were the sails are fabricated before that, for example do we know the material of the sails of Jianzhen's boat that brought him to Japan for diplomatic activities during the Tang dynasty?
Early Chinese sails were woven matting, made using bamboo leaves, other long thin leaves, thin peelings of bamboo skin, reeds, palm leaves, etc. These were reinforced by horizontal bamboo battens. Battened matting sails remained the most common type into modern times.
Cloth sails were in use by the late Han Dynasty. This is long before cotton would have been used. The two most likely candidates for sailcloth were hemp, mostly grown in the north, and ramie (Boehmeria nivea), mostly grown in the south. By the Han Dynasty, both of these fibre crops had been grown for thousands of years. Two other fibres might have been used: kudzu (Pueraria montana var. lobata, AKA Pueraria thunbergiana, with common names East Asian arrowroot, vine-creeper, etc.), and banana. Kudzu was used as a fibre plant during the Zhou Dynasty, if not earlier. The earliest records of banana fibre are from the Han Dynasty.
Some ships carried both matting sails (as their main sails) and cloth sails (to use as additional sails when running before the wind). Cloth sails would have been attractive as additional sails since they were lighter and easier to store. Some ships also used "silk sails" as additional sails - perhaps sails of some other cloth with silk embroidery, but silk cloth is possible.
Other fibres were in use by Medieval times. One 13th century shipwreck had remains of ropes - fibres of bamboo, rattan, flax, and palm were identified. (None of the sail material survived.)
do we know the material of the sails of Jianzhen's boat that brought him to Japan for diplomatic activities during the Tang dynasty?
Diplomatic and other official ships usually used cloth sails, apparently because these were seen as higher-class than matting sails: matting sails for low-class tasks such as making money, and cloth sails for more important things like diplomatic missions, and other government stuff.
References:
Needham, Science and Civilisation in China, vol 4 part 3 discusses nautical technology (and vol 5 part 9 textile technology).
McGrail, Sean, Boats of the World: From the Stone Age to Medieval Times, Oxford University Press, 2002, has a chapter on East Asian ships.