This needs to be set in a context of membership numbers for the two largest political parties. Individual Labour party membership was 419 311 in 1935, remaining in the 400-450 000 bracket through the second half of the thirties, with a large dip into the 200 000s during the war, and back up to 487 000 in 1945. Conservative party national membership figures were not collected before 1946, when they constituted 911 000. Surveys of wartime constituency Conservative associations showed patterns of falling membership due to being bombed out and financial hardship which suggests a significant dip in membership during the war for the Tories, although one cannot say precisely to what extent it mirrored Labour's. (Figures on both parties from Thorpe, 2009.)

That BUF membership figure of 50 000 as quoted on Wikipedia is from a source written in 1969. The number is not settled (Linehan, 2001) as definitive records from the time either do not survive, or else remain in BUF files seized by the British govt in 1940. Historians do generally estimate 40-50 000 for 1934 (Linehan, 2001) prior to the violence at the Olympia rally, which resulted in the Daily Mail withdrawing its support for the party, and swiftly plummeting membership down to 5000 by 1935 (Webber, 1984). Membership numbers grew somewhat towards the end of the 1930s, which Webber estimated as 22 500 nationally for 1939. When set in the context of party memberships at the time, these not large numbers.

The BUF did not run in general elections, and was instead sticking to placing council candidates in areas where it had concentrations of support. The New Party, from which the BUF was formed, but which was not at that time a fascist party (it became the BUF in 1932 after its leader, Oswald Mosley, turned to Fascism), had run in one by-election in 1931, taking 16% of the vote.

Oswald Mosley was, immediately prior to his move to fascism, a respected young politician regarded as brilliant by many of his peers. He had been a member of both the Conservative and the Labour Party, leaving Labour to found the New Party because of dissatisfaction over party failure to adopt a policy document on unemployment which he had largely authored. (This answer I wrote a few days ago refers to his pre-BUF reputation.) So the leader initially gave the party credibility and appeal to those involved in mainstream politics, in a way that someone coming from the fringe would not, and - as might any new party led by a well-known young politician who'd recently broken from the main parties, enabled it initially to attract those who were dissatisfied with existing party politics.

Unemployment and the effects of the Great Depression were serious problems in early 1930s Britain, and Mosley had previously established himself as a man with a plan on those issues.

Linehan, doing a survey of the historiography, concurs with the earlier Webber that a significant portion of the 1932-34 BUF membership was made up of dissatisfied Tories, "patriotic anti-socialist 'Rothermere fascists' [Lord Rothermere being the editor of the Daily Mail] many who whom believed (mistakenly as it turned out) that in the BUF they had found a more virile and active form of Conservatism." (Linehan, p.167)

The violence at the Olympia gathering in 1934 was decisive in decreasing the middle class membership, as the Daily Mail withdrew its backing from the BUF, and during the later mid-30s, working and lower-middle class membership became increasingly significant to the party. (Another reason for loss of support in 1934 was that Mosley had predicted a further economic crisis for the early 1930s, and by 1934, it was apparent this had not happened.) By 1939, however, membership included more disgruntled Tories who were in favour of continuing the appeasement policy.

Violence at gatherings would continue to be a problem; according to home office records, violence broke out at 46-80% of meetings each year, in the years 1934-38 (Cullen, 1993).

Patchy campaigning in working class areas was among the reasons that the BUF did not become more popular outside certain core pockets of support. At times it tried to take advantage of the fact that trade union membership had almost halved, from 8.3 million in 1920 to 4.4 million in 1933, leaving millions of workers outside the organised labour movement (stats from Pugh). An improving economic situation from 1934 was another reason for its lower appeal.

The BUF was careful to show its policies were compatible with feminism, including on matters such as women's right to work and receive equal pay, and equality within the state and political movement (Linehan, 2001). These principles were not universally accepted by establishment men of the time. There have been several studies on its appeal to women, some of whom had a Tory background (Linehan, 2001). This BBC article gives brief accounts from several people whose mothers from different social classes were involved in the BUF in the 1930s.

WWII historian Richard Overy, in The Morbid Age evokes the intellectual life of the 1930s as a time in which public intellectuals and some of the middle class who listened to them feared an end of civilisation, and that this both resulted from and fed into the increase in more extreme politics. (Eric Hobsbawm, reviewing Overy's book, points out that this was less evident in cultural forms targeted at working class audiences.) Notions of scientific racism and eugenics were mainstream among educated white people at the time; it was part of the legacy of WWII that these fell out of acceptability. Fascism could appeal to the educated middle classes from either a Romantic or Modernist / Futurist philosophical stance, or even both. These cultural trends of modernism and romanticism were also present more subtly in popular entertainment, media and aesthetics, to those who would not have actively discussed the philosophies by name.

And BUF publicity tactics sought to appeal to these cultural viewpoints, along with the use of what would now be termed as branding and celebrity, as described by Julie Gottlieb. Mosley sought to take advantage of his charisma, rhetorical ability, political reputation and photogenicity in pushing a cult of personality and fanaticism (a word which then had one usage similar to that of the current fandom) similar to that of the big film stars of the day. This glamour appealed to some, but its egotism repelled others. Posters featured consistent iconography, and campaigns were organised and made use of what technology was available - although the latter was hampered when Mosley was banned from the BBC in 1935. Gottlieb suggests that young people at BUF gatherings displayed similar behaviour as that which would be seen in future decades at pop concerts. Opponents of Mosley who went to check out rallies for curiosity or journalism also often resorted to the language of the cinema to describe the theatrical spectacle created. A range of branded merchandise was sold, including common items such as cigarettes, fireside pokers and writing paper. They tried to create a small alternative economy, having items such as uniforms made in BUF owned factories and sold in their own shop. (How all this branded gear was viewed by opponents is, however, not mentioned.)

It is arguable whether a political party that did not even feel it was in a position to contest general election can actually be described as popular, but hopefully the above outlines a few of the reasons why it appealed to its supporters.

References

Stephen M. Cullen, Political Violence: The Case of the British Union of Fascists, Journal of Contemporary History, Vol. 28, No. 2 (Apr., 1993), pp. 245-267

Stephen Dorril, Blackshirt (2006) [a large amount of the text is available at Amazon UK]

Julie Gottlieb, The Marketing of Megalomania: Celebrity, Consumption and the Development of Political Technology in the British Union of Fascists, Journal of Contemporary History, Vol. 41, No. 1 (Jan., 2006), pp. 35-55

Eric Hobsbawm, C (for Crisis), London Review of Books (2009) [Review of The Morbid Age: Britain between the Wars by Richard Overy] Available from: https://www.lrb.co.uk/v31/n15/eric-hobsbawm/c-for-crisis

Thomas Linehan, British Fascism, 1918-39: Parties, Ideology and Culture (2001).

Martin Pugh, 'Hurrah for the Blackshirts'!: Fascists and Fascism in Britain Between the Wars (2005)

Andrew Thorpe, Reconstructing Conservative Party Membership in World War II Britain, Parliamentary Affairs 62(2) · April 2009

G. C. Webber , Patterns of Membership and Support for the British Union of Fascists, Journal of Contemporary History, Vol. 19, No. 4, Reassessments of Fascism (Oct.,1984), pp. 575-606