From what i know, at first the movement started harmlessly. But then later in 70s and 80s, the whole thing became asscociate with Neo-Nazi. What the reason for this?
First it is important to understand what the skinhead culture was. The skinhead idea came from working class youth in London during the 1960s who felt that their identity was being alienated within society and wanted to gather up similar minded people of the working class and reach solidarity with one another. From the idea of working class solidarity came the components of the subculture in its fashion; expressing oneself with clothes typically associated with the working class such as doc marten boots and button down collared shirts. The culture hopped onto 60s counterculture that rejected the peace and love ideas found during the decade and was instead in favor of aspects that seemed to naturally associate with the working class and in turn their anger at their alienation. This is what originally drew skinheads to associate with ska music during the 60s as it was seen as a working class genre and was primarily seen as being made by a working class immigrant group.
The 60s came and went and with it came an evolution of the skinhead culture toward a new style of music in the late 70s that was both made by working class youth and was about being a working class youth: Oi! While ska would remain popular in the culture up to the present day, Oi! became the preeminent music to tie itself to the skinhead subculture for a couple of reasons. The first is of course that it aesthetically aligned itself with working class ideals, themselves being working class, and second is that the punk counterculture was the "next big thing" and naturally allowed a large swath of angry youth into itself to express their anger.
With the brief history of the culture out of the way we can look out how the far right fit into the picture. First, let's examine what the far right is looking for when seeking out people to recruit to their cause. Generally they want people angry at society, the angrier the better, as they will be more willing to adopt extreme strategies to solve their frustrations. These angry people usually made up the working class who have historically, at least since the twentieth century, wanted to change their position in society.
Next is how they target people. The far right favors spreading a message that speaks to the disenfranchised, those that society alienates, and their message will tell that society around them is putting them down for their class. Then they extend that their class and general position in society right now had been corrupted due to their race. That they are being pushed out of their original position in the middle and upper classes that was historically their birthright and that only by adopting far right policies can they move up from their position.
The skinhead subculture fits perfectly into being targeted by such groups and the National Front and the British National Party (both far right political parties) saw this and decided to target the subculture. At the time the far right was experimenting with using music as a tool for recruitment that was first tried by Italian far right parties with bands such as Janus but wouldn't be successful until the far right in England organized a set of concerts to spread their message.
The concerts would be called Rock Against Communism (RAC) that were playing off the name of the Rock Against Racism concerts that were also happening in the 70s. These concerts were built as a recruitment arm of the far right and tried to pull in bands that would appeal to their primary demographic. This led to the concerts which were headlined by The Dentists and The Ventz during the 70s (both punk bands without much of a following) and led to the concerts not being all too successful.
The culture heading into the 80s was commonly becoming associated with the Oi! genre of music that was getting more and more press coverage with journalists like Gary Bushell pushing the genre with articles and compilations to celebrate the genre coming into fruition. The genre was critically successful and much like the original punk wave of the 1970s there was a media rush around the genre that was increasingly becoming known for its working class and often violent attitude that was an offshoot of punk shows being prone to causing violence in and out of shows. This violence streak came to a head in 1981 with a riot during a 4 skins show that was spurred on by racial tensions between Asians in the area where the concert was being held in the mostly white and skinhead audience that was in attendance. The riot led to around 110 people injured and the media ran with the story.
The genre and subculture got tagged with starting a race riot which led to the genre growing away from the critical and media success it had had in the year or two prior but still getting decent coverage. The coverage became a praise of the music but there was an increasing focus on the violence caused by the bands following shows.
The genre finally coalesced into the neo-nazi association in 1983 when the far right finally got a band for their RAC concerts that the skinhead culture was rather into: Skrewdriver. They were one of the first Oi! bands that formed in the 70s but broke up after one record. Their vocalist, Ian Stuart Donaldson, would later reform the band after he himself adopted far right ideology and made the band one that would explicitly play music that demonstrated that ideology. The stage was set and the concert went on and was quite successful in drawing in a crowd. The far right finally got a foothold and the media began to cover this aspect developing in the culture.
The skinhead culture was targeted by the far right because their culture fit in with what the far right was looking for. From that came the race riot(s) that started from concerts that the culture was associated with and with the success of RAC concerts in 1983 that was able to draw in and convert people to far right ideology the association of skinheads with neo-nazis stuck.
While the skinhead culture did get hijacked by the far right, that does not mean that the entire culture transitioned into it. The bands and original skinheads of the 60s and 70s were largely apolitical and some even adopting a staunch anti-fascist statement (ex. The Oppressed). The far right skinheads often mingle among themselves and mostly intermingle with themselves rather than the skinhead culture at large. You'll see this culture even mocked by the apolitical or left leaning skinheads as the far right are labelled "boneheads".