Darwin's choice of a book in general was a result of his belief that his theory would require a mountain of evidence and reasoning for anyone to take it seriously. The Origin of Species wasn't even meant to be the real book — it was the "abstract" for the real book that he was dashing off quickly in order to secure his claim to priority after realizing he was going to get scooped by Wallace. (He never wrote the "big book" — though one can imagine combining several of his books into one giant tome and having it stand in for the omnibus he was planning.)

For Darwin, it wasn't just the idea that mattered. You can express the basic idea in a few sentences. It was getting people to take it seriously. He took very seriously the cases of Lamarck and Vestiges of the Natural History of Creation, both of which presented evolutionary ideas but were so un-compelling that they were not taken seriously by the British scientific elite. He was determined to produce something that would be more successful, and he felt that accumulation of evidence was what was going to do the trick.

Why'd he write it the way he did? I don't have any indications that he wrote it in in a deliberately readable fashion. That was just how Darwin wrote. He was emulating the Romantic style of writing, which was that of his heroes, notably Alexander von Humboldt. He is writing for an educated crowd, but it is not just a specialized crowd. But he does contain specialized discussion in the book, especially in the footnotes. I would not characterize Origin of Species as "pop science" — I would characterize it as a work that was written in a particular idiom (Romantic grand scientific narrative) that happens to be more accessible than, say, the sorts of quantitative studies that would come later. Darwin was a Romantic gentleman scientist and it shows in his approach to writing, especially when you compare it to other contemporaries or later scientists.

So the interesting phenomena here is that "the masses" wanted to read Darwin's scientific book, not that he wrote for "the masses." In this Darwin himself credited the ill-fated, anonymously-written Vestiges of the Natural History of Creation (1844). Vestiges was a very "pop sci" book of its time, deliberately written to be accessible to an increasingly literate British publish who could afford increasingly cheap books. It caused a sensation for its broadly cosmic and Lamarckian evolutionary argument, even if it was ignored by most professionals. Darwin credited it for raising the awareness and interest of the British public in this kind of argument, and "priming the pump" for some of these ideas (even if he also was embarrassed by it, because it was definitely not what he was trying to accomplish).

Darwin's publisher saw an opportunity in Darwin's book (and Darwin himself had already produced a minor best-seller with his book on his Beagle voyages), and hyped it at various trade shows. It got taken up into Mudie's Lending Library, a sort of Victorian Netflix (the original DVD form) for books, and through there had a very large public circulation indeed. This was not really Darwin's doing; he just wanted the book out there, and he cared far more about how other specialists read his work than he did the general public.

Darwin did write many journal articles in his lifetime, though there were always on much "smaller" topics than those he covered in his books. The history of the rise of prominence of the scientific journal over the course of the 19th century is another topic altogether, though some of Darwin's contemporaries (like Huxley) played a major role in this. The real question I think you are asking, though, is why scientific, book-length monographs decreased in importance over the 20th century, and became less common as a research output. I don't have an exact answer for that (if it is really true) but I think that the increase specialization and research productivity expectations play a big role in that (it takes much longer to write a book than it does to write a journal article, and scientists generally get grants and tenure based on their raw research productivity these days).

On Darwin's writing of the Origin, and the influence of Vestiges and other publishing consideration, Browne's Charles Darwin, Vol. 2: The Power of Place is excellent. On the history and reception of Vestiges in Victorian culture, see Second, Victorian Sensation.