I answered a little bit [here] but I'll touch on your other parts.

The tartan choices came out of the diaspora and a bit of marketing. Many of the clan tartans started to have a standard one in the early 1800s. More of them started to appear in the early twentieth. They were really a fantastical element as clan tartans did not really have the same connotation pre-1800s as they do now. That craze started after George IV's visit to Scotland in 1822. It was a pretty big deal since no monarch had visited Scotland for a long time at that point. That Scottish or Highland Diaspora also has it's roots in the 18th century in around 1782 with the repeal of the Dress Act. The Highland Society of London brought in a lot of Scots to commemorate their heritage and would feature bagpiping competitions and kilt wearing. George the IV made the kilt the national dress of Scotland, which took away the distinction of being Highland garb as it had been for centuries. Now people were really wanting to be a part of it. They wanted connection with their heritage, and that came with clan tartans.

The creation of clan tartans in the 19th century sprung from the King's visit. Many were created for the pageantry. In 1842 the Vestirium Scoticum was published by the Sobieski brothers. Hilariously they claimed to be grandsons of Prince Charlie Stuart. It's not at all true and was much in part to drive the romanticism in their forged ancient histories of clan tartans. They claimed the book to be a copy of a 15th century manuscript and sold as such. Many of the patterns though were used as official tartans for clans. The tartans illustrated inside do not have any historical basis. One of the earliest reports of "clan tartan" was in the early 1700s by Martin Martin (Not joking, that's his name) depicting regional differences among tartans from certain islands. From my understand, this came from the technological limitations of the looms used. The looms used to make the ells of tartan (yardsish) were around 27" wide so fairly narrow. They're not going to just run a small batch but run literally hundreds of ells because a patron liked the design and paid for it. There are some surviving examples of tartan worn in the 18th century but it would be stretch to call them clan tartans as we would think today.

The modern clan tartan creators ran with the romanticism of the clan idea. The overall clann with it's many septs could lead to different versions of tartans - more to sell. Again most have no historical basis, save some stemming from the Sobieski book. Most were created to have connection between clan society members by clan chiefs. It's a nice sentiment and I think it does create a family like appeal to it. In the middle of the twentieth century, tartan weavers started to create variations of tartans such as modern, weathered, and ancient. The names really do not mean anything but how they are dyed. Modern is using the pigments of a brighter dying plant, weather is to depict something buried in peat and brought up after many years, and ancient to look...well, old. It does not mean it was worn by Scots in 30 AD. It's all marketing.

For the Irish kilts, it also came from the Irish diaspora and the Celtic revival of the late 1800s. I do not have much primary sources for this and I hope someone can add to this. The Celtic Revival in Ireland looked to created a national dress such as Scotland's and decided that the kilt would be the most rational one due to the Celtic relations between Ireland and Scotland. However, they did not use tartans, instead they decided solid colors would be the ideal Irish kilt. Blue and saffron are the two suggested early on, green was used as well. Today you can see armed forces of Irish regimental pipebands wearing saffron kilts. The Irish nationalists of the revival chose saffron from the historical popularity of it centuries before. Many shirts and clothes were dyed with it and remained a staple in Irish fashion for many decades. There are some Irish tartans that have historical basis. The Ulster tartan was devised from a piece of cloth found in a bog. The rest were created in the twentieth century. There is a similar reason for the Welsh kilts with a Celtic revival in the twentieth century as well.

Two sources of reading would be Albannach.org, although it may be a bit dated now since Matt Newsome hasn't had any new research in years.

Peter MacDonald, kilt historian, is another. He is ongoing and has some really cool stuff on his site.