What do you mean by a "modern computer"? It's unclear. For example:

You could mean a computing engine capable of being programmed to perform different tasks. In such a case Babbage's machines (the Difference Engine) would count.

Or you could mean an actual working computer powerful enough to have an impact on society (to render possible previously impossible or at least incredibly time- and labor-intensive tasks) in which case punch card or vacuum-tube machines might be what you mean by "modern".

Or by "modern" you could mean a computer that uses transistors to miniaturize circuits and hence bears a working resemblance to the machines we use today.

Or you could mean ... but you get the picture. What are you asking about?

One way you could define "the idea of modern computers" is "a theoretical machine that is Turing-equivalent", in which case the earliest (but not most important) one would be Charles Babbage's work on the Analytical Engine starting in 1837. Usually people talk up his earlier Difference Engine, which was essentially a very large mechanical calculator for solving polynomials, and doesn't really correspond to a modern computer at all. The Difference Engine project failed due to budget limitations and manufacturing technology available at the time. This meant that when Babbage proposed his Analytical Engine, he was unable to secure enough funding to complete more than a small portion of it, and his work was largely forgotten until the late 1940s.

In the meantime, other theoretical machines were developed during the exploration of David Hilbert's Entscheidungsproblem. This asked for an algorithm that, given a statement of first-order logic, could decide whether it is provable from the axioms or not. The first step towards solving the Entscheidungsproblem was to define what exactly an algorithm was; without a formal notion of an algorithm, you couldn't prove that there was no such algorithm. In 1936, Alonzo Church and Alan Turing independently published two equivalently powerful models of computation, the lambda calculus and the Turing machine (called the "a-machine" in Turing's original paper). Having created these models, they (independently) put forth the Halting Problem (given a program, does it halt?) as a form of statement that no algorithm could answer correctly. The proof of this is a little roundabout. First, assume that there is some algorithm to decide the Entscheidungsproblem. Then, you can construct a machine that encodes a description of itself into first-order logic and then runs the algorithm to decide whether it halts. Finally, if the algorithm said it halts, it runs forever, and if not, it halts immediately. Any algorithm will be incorrect about the particular machine designed to defeat it, so there can be no algorithm that always correctly solves the Entscheidungsproblem.

The Turing machine and lambda calculus were the theoretical models of computation that early computer scientists focused on. As I mentioned above, Babbage's Analytical Engine was mostly forgotten at the time. Even once it was rediscovered, it was never really seen as anything more than a historical curiosity. Babbage never really specified the interface of the machine. Instead, his notes focus on the implementation details of specific subsystems using mechanical components. On top of that, the Analytical Engine is terrifically complicated to reason about. As a result, the theoretical side of computer science never really looked into it. On the implementation side of things, electro-mechanical computers had proven to be far more effective than their purely mechanical predecessors, so no one considered building an Analytical Engine with any seriousness. It wasn't until Allan Bromley's 1982 paper that any modern computer scientists really dug into Babbage's work, and even today there are aspects of its design that aren't clearly understood.