Detective fiction can be arguably traced to Edgar Allan Poe's "The Murders in the Rue Morgue" (1841), which introduced C. Auguste Dupin as a non-professional, highly intelligent gentleman who having found a mystery that baffled the police, set about solving it. Poe's character set the stage for the development of other fictional detectives, notably Sir Arthur Conan Doyle's Sherlock Holmes, who entered the scene in "A Study in Scarlet" (1887), and Agatha Christie's Hercule Poirot in The Mysterious Affair At Styles (1920).

All three of these figures were, more or less, amateurs who assisted the police in their investigations - but there were several other kinds of popular detective fiction. In the late 1800s especially Pinkerton Detectives and lawmen were popular fair for dime novels with titles like The New York Detective Library and Old Sleuth Library. These stories gently mixed the central idea of a mystery, with the cerebral solution, with elements of action and true crime.

The dime novels gave way to pulpwood magazines around the turn of the century, and after World War I pulps exploded in popularity. One of these new pulps was Black Mask, founded by H. L. Mencken and George Jean Nathan in 1920. Initially intended as "the best stories available of adventure, the best mystery and detective stories, the best romances, the best love stories, and the best stories of the occult" and early issues featured very traditional stories of private consulting detectives in the Dupin/Holmes/Poirot mold... but in June 1923 they published "Knights of the Open Palm" by Caroll John Daly, starring Race Williams who is often considered one of the first "private eye" character in American fiction.

Unlike Holmes & co., Race Williams was not a cerebral sleuth; he was essentially an adventurer-for-hire. To quote the character:

The papers are always either roasting me for shooting down some minor criminals or praising me for gunning out the big shots. But when you're hunting the top guy, you have to kick aside--or shoot aside--the gunmen he hires. You can't make hamburger without grinding up a little meat.

This was a fundamental change from Conan Doyle's erudite, gentlemanly consulting detective - but it was really only a development of the American dime novel, with the focus on action, realism, and thrill emphasized. What would come later, and would largely define "hardboiled" fiction as we know it, is disillusionment, cynicism, and detachment.

Detectives flourished in the pulps, with magazines like Detective Fiction Weekly, Detective Story, Action Detective, Dime Detective, Nickel Detective, Black Book Detective, Double Detective, Triple Detective, and Thrilling Detective - not to mention Spicy Detective, Strange Detective, Crime Busters, and dozens of mystery pulps as well. There were occult detectives in Weird Tales and crimebusters in the weird terror pulps - but they weren't all hardboiled; they were often a mix between Sherlock Holmes and Race Williams, balancing physical action versus cerebral sleuthing.

Joseph Shaw took over the editorship of Black Mask in 1926, and re-organized the magazine to focus on crime fiction. One of his cornerstones was Dashiell Hammett, a retired Pinkerton detective turned pulp writer who created the immensely popular figure of the Continental Op, a nameless operative for the Continental Detective Agency, as well as Sam Spade, the star of the popular novel The Maltese Falcon which was serialized in Black Mask in 1929 and collected between hardcovers in 1930. In 1931, a film version was released, followed by another adaptation in '36, and finally the famous version starring Humphrey Bogart in 1941 - the last shot in the distinctive film noir style which would later become almost synonymous with hardboiled films.

Daly and Hammett were not on their own; aside from a legion of imitators in the pulps and professional writers like Earle Stanley Gardner, in 1933 Shaw attracted Raymond Chandler, whose hardboiled character Philip Marlowe would provide Bogart's other definitive hardboiled detective role. Chandler's character borrowed from the realism of Dashiell Hammett and expanded it with a kind of cynical, detatched philosophy, which he famously summed up in his essay "The Simple Art of Murder" (1950):

In everything that can be called art there is a quality of redemption. It may be pure tragedy, if it is high tragedy, and it may be pity and irony, and it may be the raucous laughter of the strong man. But down these mean streets a man must go who is not himself mean, who is neither tarnished nor afraid. The detective in this kind of story must be such a man. He is the hero, he is everything. He must be a complete man and a common man and yet an unusual man. He must be, to use a rather weathered phrase, a man of honor, by instinct, by inevitability, without thought of it, and certainly without saying it. He must be the best man in his world and a good enough man for any world. I do not care much about his private life; he is neither a eunuch nor a satyr; I think he might seduce a duchess and I am quite sure he would not spoil a virgin; if he is a man of honor in one thing, he is that in all things. He is a relatively poor man, or he would not be a detective at all. He is a common man or he could not go among common people. He has a sense of character, or he would not know his job. He will take no man’s money dishonestly and no man’s insolence without a due and dispassionate revenge. He is a lonely man and his pride is that you will treat him as a proud man or be very sorry you ever saw him. He talks as the man of his age talks, that is, with rude wit, a lively sense of the grotesque, a disgust for sham, and a contempt for pettiness. The story is his adventure in search of a hidden truth, and it would be no adventure if it did not happen to a man fit for adventure. He has a range of awareness that startles you, but it belongs to him by right, because it belongs to the world he lives in.

If there were enough like him, I think the world would be a very safe place to live in, and yet not too dull to be worth living in.

You can see how far this is, both from the cerebral but detached Sherlock Holmes and the bloody-minded Race Williams. Chandler's sketch of a hardboiled detective is an everyman, world-weary, but not beaten, a hero that might appeal to men and women who had lived through at least one and possibly two world wars.

Which is basically how the hardboiled detective emerged - as the idea of detective fiction adapted to popular tastes and trends, which themselves shifted over time. Black Mask peaked in sales in the 1930s, but declined and the writers moved on - to Hollywood, for Hammett and Chandler; to the other pulps and comic books for others, novels and slick magazines for still others.