They're known as the Pan-African colours, and they're a reference to Ethiopia, which largely avoided European colonization. Ethiopia adopted a red-yellow-green tricolour in 1897, and when other African countries started gaining their independence they deliberately borrowed that same colour scheme.

The Rastafari community uses those colours because of their worship of the Ethiopian emperor Haile Selassie. One of the most commonly used "Rastafari flags" is the former flag of Ethiopia.

There's also second set of Pan-African colours: red, black and green. Those are the colours of the Pan-African flag, designed by the UNIA in 1920. There are a few possible explanations for where those colours come from, but one of them is that Marcus Garvey erroneously believed that they were the colours of the Ethiopian flag. All four colours are now widely used to represent Africa, as well as certain countries in Oceania and the Caribbean.

The red, yellow, and green color scheme is derived from the flag of Ethiopia, which first came into official use around 1897. This was shortly after the First Italo-Ethiopian War, in which Ethiopia defeated Italy and subsequently had its independence recognized by most of the major powers of Europe. At that time, Ethiopia and Liberia were the only African states that remained independent of European colonization (although Liberia was originally a US colony, it had gained independence in 1847).

The colors of the Ethiopian flag were one set that became popular among Pan-African and decolonization movements of the 20th century, as they came to represent African independence. The other set of colors that would become associated with Africa during that time was red, black, and green. This set was more specifically intended to be the colors of the Pan-African movement, and was promoted by Marcus Garvey and adopted by the UNIA-ACL for that purpose in 1920 (point #39). Upon gaining independence, many African states adopted one or both of these sets of colors for their flag design.

As for the Rastafari usage of red, yellow, and green, that's due to the movement's worship of Haile Selassie, emperor of Ethiopia from 1930-74. Additionally, the Ethiopian flag featuring the Lion of Judah that was associated with the Ethiopian monarchy has become a major symbol of Rastafari culture. Sometimes black is also incorporated along with the other three colors, likely due to the tremendous influence Garvey and Pan-Africanism had on Rasta ideology.