I'm going to try and break this down a bit. If you want me to elaborate on something etc. just let me know:

How powerful were local Soviets (i.e. workers' councils)? Did they provide a genuine voice for communities at higher levels, or were they basically functionless placebos?

The power and importance of local party organs is kind of a question of what you expect them to achieve. For the CPSU, the local party organs were essentially about mobilisation with some role in the management of matters in their local areas. In this I think we could say there were fairly powerful. Local party organs were often central to members social lives, their place of employment, housing etc. and for an array of reasons (from ideology, to be generally in favour of some policies to the local party's coersive power) people were receptive to their message. In economic matters some local party organs had great influence over local enterprises, but were often in competition with the state/ministries who formally ran those industries and often had some overlap in other areas. For example your housing might be built under the auspices of the local factory being built, which makes the ministry responsible for the housing even if the local party notionally allocates it. I'm not sure I would go so far as to call them placebos, though some of their function was at times to have the rank and file blow of steam essentially.

They formally had a role in the formulation of higher end strategic policies but that was more of a mixed bag - issues coming up from the local party level could eventually make their way up to the top, but their ability to influence them was mixed. They couldn't exactly force the Central Committee to make policies, it would be more of a 'lobbying' thing, and even then that could be more of an informal process rather than something done out in the open.

To what extent could non-party members participate in the political system, and how easy/difficult was it to become a party member?

Non-party members didn't really have much say. They could vote for the Congress of People's Deputies, which was the equivilent of the government parliament, but the importance/power of that structure was minimal - the party set policy and the economic apparatus (sovnarkom, the ministries etc.) could essentially do its own thing.

The ease of joining the party varied quite a bit over time. IIRC to join the party an existing member had to vouch for you, and there was a candidacy process so if you weren't up to scratch you could be thrown out early in the process. Party members were supposed to embody certain values, be politically active and ideally come from an appropriate background (i.e. being a factory worker instead of a former member of the nobility). But in the post-revolution period joining could be fairly easy because the party was so desparate to mobilise people and have people who could do practical work.

By the 1930s though there was a lot of push back against party members whose participation/work was poor, or who weren't meeting expected standards. There was also a lot of concern about exactly who was being let in, and this being accountable - in some areas record keeping about membership was dreadful and the local party would lose track of people because they moved to other areas, or indeed conduct no background checks on people who arrived in the area. There were some purges (not in the sense of the later Great Terror of 1937-1938) where there were mass reviews of existing members, re-issuing of party cards and ideally investigation of their past involvement/behaviour. These administrative purges though didn't always go to plan though, as local party organs were not always keen to shed too many members, and political (or actual) bosses were inclined to protect their people where possible.

How easy/difficult was it for those in positions of power (mostly Stalin, but also other top officials) to implement their agendas? How important was the "top-down" structure of the Communist Party?

This is actually a subject of a lot of debate. I would say it was a mixed bag. The example above about the membership purges and how the local apparatus could undermine or fail to deliver in the intended way highlights when the two forces could 'clash' and how even if they wanted something the top levels of the party might not get what it wanted exactly.

Basically if the 'centre' wanted things to happen a certain way, they could make it happen, it was just a question of them knowing about the issue, caring about it and having the resources to intervene. For example for serious interventions someone from the Politburo would arrive in town, investigate the problem, quite likely publically criticise and/or fire some people, and possibly divert resources to ensure the project/job in question is done as intended. The threat of central intervention was enough to ensure the local party, enterprises etc. broadly speaking behaved.

But the Soviet Union was a big country with lots of problems and stuff to do, and this meant that the centre didn't always have the time or resources to deal with every problem. Infrastructure and distance also meant they didn't necessarily know what was really going on - especially if the local structures were willing to mislead them. The local party or state apparatus could have a lot of leeway in things simply because there was a lack of specific direction etc. and other than being told that they must meet targets, the centre didn't really care how this was achieved so long as they didn't make too much of a mess.

Finally, and this is admittedly pretty broad: How much power/leeway did the NKVD and KGB really have? That is, how much of a repressive force were such policing organizations? Were they as formidable as generally portrayed?

This depends a lot of era. So for example under Yezhov in the windup to the Great Terror (or by its alternative title - the Yezhovschina), the NKVD had a lot of leeway in investigations and essentially smashing through any protection someone might seek. But during the 1st Five Year Plan the preference was to keep the police out of anything that might impede industrialisation - local bosses or leaders like Ordzhonikidze for example could protect people with dubious pasts or dodgy behaviour if they needed them to meet targets. At some times the formal police (MVD) and judicial apparatus had some capacity to balance the NKVD. Post-Terror the NKVD was defanged a bit and had a few re-shuffles of structure, though at times it could have a fair bit of power.

What types of books/documents would be most useful to study? Most of what I've written above comes from:

• Stepehn Kotkin - Magnetic Mountain (good overview of the party/state tensions during industrialisation, has a big section of the later purges, policing, the NKVD etc.)
• J. Arch Getty - The Origins of the Great Purge (very much about structures, a bit outdated and contested, but I think a good overview of the administrative purges and what local party politics could be like - the later book The Road to Terror with Getty and Naumov is better in terms of the origins of the terror etc... but less specific about local power politics.)
• Sheila Fitzpatrick - Everyday Stalinism (this is more so about what it was like to live in the era than about structures and such but does a good job of explaining the role of the party in everyday life and how people felt about it all)

The most accessible way to start studying the Stalin period probably is biography. I would recommend starting with a more middle of the road work like Robert Service's Stalin. Simon Sebag Montefiore's biography (Court of the Red Tsar) is decent but perhaps leans more towards a populist history. Stephen Kotkin has just released the first volume of a biography on Stalin - I haven't read it yet but based on Kotkin's other works I would suggest it might be better to understand the general history (and orthodox interpretations) of the period first, before getting carried away with Kotkin.

Off the top of my head a more totalitarian/top down vision of Soviet politics and the terror would be Robert Conquest's The Great Terror. That however is less so focused on local power politics so might fall outside your area of interest. I'm actually keen to see if there is a more current (and academic) take on the totalitarian model though because Conquest did most of his work before the archives opened, and was pretty entrenched in his positions by the time they did open.

Hope that helps and made sense.