In the 1974 referendum, the vote was overwhelmingly in favour of independence (99+%) everywhere except for in Mayotte, which was majority against it. What were the factors that led to this difference in voting?
That is a very good question. I have always been interested in the topic and wrote a paper on it, so I'll try to address it.
In 1958 with the new Constitution and then in 1961 with a law organizing the internal autonomy of the Comoros (according to some this is the first time the French government considered the Comoros as a whole instead of the islands separately) a double administration was put in place in the Comoros : on one side you got the French government, represented by a High administrator and on the other side there was a local representative government elected by Comorians, sort of like what's in place in New Caledonia right now. The High administrator kept a lot of power (mostly regalian powers) but devolution was pretty advanced.
Mayotte was less populous than the other islands and therefore only had 4 representatives a the local assembly. There was a widespread sentiment in the local population that the new capital in Moroni (the capital used to be in Dzaoudzi, on the island of Mayotte) did not care about the interests of Mayotte. Therefore the population tended to appeal more to the French representative than to the local government with their problems or requests. The French High administrator was also of the opinion that the system did not let the interests of Mayotte be heard. In 1959 he modified the voting system in order to go from one electoral district for all the islands to a district per island.
So this francophilia that would become typical of Mayotte mostly started as an opposition to the local government in Moroni and its attitude toward Mayotte. And already at this time there were people calling for Mayotte to become a département, something you could hardly see on the other islands.
The 1974 consultation (it wasn't a referendum per se) was voted in the electoral districts drawn in 1959. But before discussing it, you should know about the Mouvement populaire mahorais (MPM). The MPM was a party created in the early 1960s that was in favor of a département in Mayotte. The MPM took as example several movements in La Réunion, which has a tradition of being very loyalist to France. It tried to tell the population that their interest lied in a continuation of the union with France. Or at least that was the main line of party since others in it had diametrically opposed ideas.
Anyway, building up to the 1974 consultation the MPM and other parties were very active in Mayotte and vocally defended the idea that being part of France, and eventually attaining the status of département, would offer a protection to Mayotte that the independent Comoros could not offer. Leaders in the MPM also realized that staying with France would eventually allow them to get elected in the Senate or National Assembly, which would make their influence as the local "big men" much more important than if they were to be part of a state of the Comoros dominated by Anjouan and Ngazidja.
In the end the movement to reject independence in Mayotte picked up a lot of traction and eventually led to a victory in the consultation. It was mostly peaceful but it now seems that in the weeks preceding the referendum violence by the MPM was reported. The existence of an armed branch of the MPM is also disputed nowadays but I really don't know enough about that. Mayotte did end up becoming a département in 2011.
My main sources (French only) are not too dated but research in this area is really barely scratching the surface, as the Comoros have never really been a big topic of interest for historians and political scientists alike: