Were they pre-poisoned a head of time, with something dry yet toxic, or did they dip it in something right before shooting?
Poisoned arrows depend on multiple factors: bow and arrow construction, toxicological knowledge, access to resources, etc. Poisoned ranged weapons from antiquity were very different than what Native Americans used. It's also impossible to pinpoint the "first" poison since a lot has been lost to time, and there were probably tribes who used all sorts of methods that went unrecorded. However, thanks to a project I recently finished I have some knowledge of arrow poisons spanning early history in different parts of the world.
Perhaps one of the first, most wide-spread source of arrow poison comes from snakes. Africa, Asia, and the Mediterranean had a myriad of poisonous snakes and vipers which Greco-Roman writers noted were in use by a variety of groups including the Gauls, Persians, Getae, Slavs, Armenians, Parthians, and Dalmations. The famous Greek geographer Strabo wrote about a poison from the Soanes of Caucasus so volatile that the fumes were toxic, and Silius Italicus listed various groups in modern Northern Africa that use 'em. Early serpent poisons were made by dipping the arrowhead in the venoms, which were sometimes heated together with secondary components. One of these secondary components was strychnos, a poisonous tropical plant that grew in Africa.
In China, the ancient poison of choice was Gu, a process that was less than scientific: a whole bunch of venomous animals were dumped together into a pot, uncooked. The animals ranged from scorpions to snakes to frogs to spiders. Then they'd open the pot after a day or so, and whatever animal was left alive last was considered the most lethal. Often the remaining animal had eaten some/all of the rest, so it had a myriad of poisons in its body. Then it would be processed into usable poison. This also required dipping darts and arrowheads into the solution.
Not all poisons were dipped, however. Poisons around Assam / Burma in East Asia used a variety of plants like antiaris, strychnos, and strophanthus with other berries and clays to make a smearable paste. This paste was then applied behind the arrowhead, instead of coated all over and dried. These toxins were exceptionally lethal and had a long shelf life—the Pitt Rivers Museum discovered a poison used in Burma against the Karen tribe as lasting up to 1300 years.
Another famous poison is "curare." Basic internet searches will try to give curare a singular definition, but it is in fact a catchall term for poison that was misunderstood for centuries by Spanish Conquistadores during their campaigns in South America. Curare meant any poison from that region, but there were in fact dozens of different kinds. Pinpointing with accuracy is difficult, as stuff published by the Spanish (eg. Raleigh's Discovery of the Large Rich and Beautiful Empire of Guiana) is often fraught with self-aggrandizement and outright fiction. However, other explorers that came later managed to find subdivisions in curare, like E Bancroft who wrote "Essay on the Natural History of Guiana and South America" in 1747. In it, he details the Woorara poison made by the Ticuna tribe:
These raw materials were mixed together over an open fire into a pitch. The tribe used palm leaves to scoop up the mixture and smear it on arrow and spearheads. As time went on, the term took on more specificity, but even Rudolf Boehm's attempt to define the term in 1895 led to three major categories: bamboo curare, calabash, and pot.
Finally, William McKnight detailed a Native American process for making poison arrows. They would leave out animal livers to decompose, then fill the rotting pouch with rattlesnake venom. The decomposing animal organ would add bacteria and the venom added the poison. I don't know the symbolism by using livers specifically, because McKnight doesn't elaborate.
Maybe someone else can shed insight on when the arrows were made. I don't know if they were made the night before, weeks in advance, or hours before conflict. I'm sure that varied by region and place in history's timeline, too.
tl;dr
Most poisoned arrows were dipped into toxic solutions made by whatever poisonous plants or animals populated the nearby area. Pastes were less common, but they did exist (and weren't specific to East Asia). The term curare doesn't refer to a single poison.
Sources:
Gray, TC. "The Use of D-Turbocurarine Chloride in Anaesthesia." (1947). Ann R Coll Surg Engl. 191-203.
Bancroft, E. Essay on the Natural History of Guiana and South America 1769, pg 281.
"Poisoned Arrows from Assam and Burma" Victoria and Albert Museum. Web. http://www.vam.ac.uk/content/articles/p/poisoned-arrows/
Leafloor, Liz. "Poison: The Good, the Bad, and the Deadly" Ancient Origins. 4 Oct 2014. https://www.ancient-origins.net/ancient-technology/poison-good-bad-and-deadly-002165
Mayor, Adrienne. "Chemical and Biological Warfare in Antiquity." Science Direct. Republished from "History of Toxicology and Environmental Health" 2015. Web. https://www.sciencedirect.com/topics/medicine-and-dentistry/arrow-poison
McKnight, James W. Jefferson County Pennsylvania, Her Pioneers and People 1800-1915. J.H. Beers and Company, 1917.