This depends on who you ask. Among theorists their ideological tendencies (pro or anti) often color the issue a great deal.

Now, from what I understand you're asking specifically about the role of the modern State in the formation of Capitalism. I am most familiar with Marxist (historiographical) analysis of the subject but please do not take explanation for outright endorsement. I do want to take a moment to recognize that the question is a well formulated one since you recognized that we're talking about the 'modern' State, which recognizes the fact that the State cannot be understood as something eternal with slight variations but rather something which takes concrete forms depending on its historical context. You also recognized that this question carries the underlying (and huge) questions of what the State and Capitalism are. That's absolutely right as well! Most people don't seem to deconstruct the premises of their questions in this way, so you deserve recognition for doing so.

The State is often understood as an organization ruling over several key institutions which monopolizes violence and participates in the economy to the benefit of specific interests within a society. Beyond that… Consensus kind of breaks down.

As for Capitalism itself, the traditional explanation of its nature is a ‘socio-economic formation’ (ie. system) characterized (as is typical for Marx) by the relations of production which are specific to it. In the case of Capitalism that means the exploitation of the proletariat (workers without ownership of the means of production, as opposed to artisans and small farmers who at least owned something) by the bourgeoisie through the latter’s ownership of the means of production, resulting in the extraction of surplus value (value produced by a worker’s labor but not remunerated by his/her employer) to the benefit of the Capitalist. Tons of other definitions exist. It’s still hotly debated.

Marx himself never offered a coherent and systematic analysis of the role of the State in any society, even mature 19th century Capitalism. The most commonly used explanation of the role of the modern State within Capitalism is that “the executive of the modern state is but a committee for managing the common affairs of the whole bourgeoisie.” (Karl Marx and Friedrich Engels: Manifesto of the Communist Party, 1848).

From this you get the seeds of the instrumentalist role of the State, which eventually mature into Ralph Miliband’s The State in Capitalist Society (1962, multiple editions) which presents what we today call Instrumentalism. In essence, the Instrumentalist position (within the Marxist theoretical tradition) is that the State is a tool for the advancement of the interests of the dominant class. In a Capitalist society that class is of course the bourgeoisie. Now, Miliband’s book and later debates with another major Marxist theorist, Nicos Poulantzas, are interesting, but also invited some important criticisms. For one, if the State is just an instrument, used (implicitly through conscious design) by the dominant class, how does that class decide what’s in their class interest? How do they generate consent? Meetings? How does this theory explain State policies which go against Capitalist interests? How does it account for the fact that Capitalists don’t necessarily rule directly, yet their proxies already seem capable of advancing their interests? Basically, the essential criticism of Miliband is that his theory requires too much conscious design behind State policies and fails to develop an organic theory of the State that is largely conspiracy free.

Running contrary to this theory is Nicos Poulantzas, a Greek Marxist theorist who advanced Structuralist theory (which I’m slightly less familiar with and which is notoriously Byzantine). The basic idea is that the State is simple one kind of social structure, parallel yet connected with cultural and (private) social spheres which organically reproduces and advances the interests of the dominant class in a given society. Basically, think of the State as being a series of institutions/ideas which by their nature (being grown in the midst of Capitalist society) is structured in such a way so as to naturally benefit Capitalists in such a way that needs no conspiracy or organization to keep it together. Examples of what this means in practice would be the way in which going into politics has traditionally required personal wealth, connections with other wealthy individuals who (due to your policies, ideas, and class background) would be naturally inclined to help you, and the blessing of the authorities to function without harassment (since radical political movements can have police agents mixed in, be the object of operations intended to disrupt it by domestic intelligence services, can be the victims of direct police repression, etc.).

Like Miliband’s theory, Poulantzas’ was extremely difficult to prove empirically. More so, in fact, since while Miliband’s hypothesis suggest easy empirical investigations into who runs the State, what class they belong to, what ideology they hold, what organizations are used to develop consensus between a class, etc., Poulantzas’ Structuralist analysis is difficult to impossible to prove empirically, especially given the author’s rejection of Miliband’s attempt to analyze individuals and their actions as somehow representative of a class. For Poulantzas a class can only be understood at a level of meta-analysis. Critics were uncomfortable with what was understood to be a sort of declaration of faith that the system would function a certain way without any way to confirm it or give coherent answers to why, then, so many concessions had been achieved by working within the system.

In the 1970s Immanuel Wallerstein and others began to include elements of Marxist analysis of the State within their broader project of World-Systems Analysis, with the analysis of the role of the State analyzed not just in terms of transformations and class struggles within the State’s borders but also as a part of the broader Capitalist World-System. Wallerstein walks the line between Miliband and Poulantzas, in addition to providing ideas of his own, by seeing the State as a tool which individuals living within and outside of its borders tweak in order to advance their own interests. We’ll get into this a bit later, when we talk about concrete answers.

On this debate and other related issues on the problem of Marxist State theory I recommend Clyde W. Barrow’s Critical Theories of the State: Marxist, Neomarxist, Postmarxist (University of Wisconsin Press, 1993).