Abstract group theory came first. Group theory first emerged from the study of the roots of polynomial equations of various degrees and the way that permutations relate the roots to one another, forming a "group". Galois, Cauchy, and Abel, all working in the early 1800s, created the foundations of what would eventually become group theory through their work on polynomial roots and later modular arithmetic. In the 1870s and 1880s group theory matured into its own "discipline" within mathematics, with enough foundation and enough mathematicians working on it to rightly be described as a cohesive subject. This was also when it began to filter into textbooks and University curricula. This was also about the time that the connection of group theory to geometry and especially to symmetries started to be developed, and through the late 19th century group theory matured into the form we are more familiar with. (Note that I'm glossing over the details of abstract algebra and the "reinvention" of mathematics in the late 19th century here since it's somewhat tangential to your question.)

Through the turn of the 20th century and just after group and symmetry theory began to see practical application in the development of quantum mechanics, the invention of crystallography, electron orbital theory, cryptography, etc.