Adapted from a couple of previous answers on the topic:
The first part of the answer is simple as the Marshals weren't relegated at all; what you describe was the main portion of their job from the Early Republic onward.
Why was this? The position was a patronage one, and up until the end of the 19th century one of the more surprising things about most patronage positions was that they were paid on a percentage of revenue obtained. So if you were a postmaster (by far the most common patronage position), you'd get a cut of every stamp sold and letter delivered. If you were a lawyer presenting a Civil War disability claim, you'd take home a hefty sum every time you filed on someone's behalf. And if you were a US Marshal, you'd get paid some for arresting and extraditing, but your real money maker was in serving paper.
As such, investigating was something they had neither financial interest in pursuing nor any particular expertise in, and this began to be a significant problem during the late 1800s given the sheer magnitude of fraud taking place. Also, it's worth pointing out (credit to the removed comment for reminding me) that by the Civil War marshals were generally seen more as functionaries of the judicial rather than executive branch despite their appointments coming from the latter; an example of this can be seen in a previous post about Roger Taney's orders to 'his' marshal from the bench in Ex Parte Merryman.
So there is essentially only one Federal agency that has any real investigatory experience during that time period, and that's the Secret Service - except when it does so, it is acting without any law providing them that authority. For that matter, even their post-McKinley role in protecting the President is assumed rather than mandated; for something like a decade, Congress essentially allows them to do so with what passes as a wink and a nudge while being aware they need to pass legislation eventually.
Unlike Presidential protection, though, the investigatory role is controversial and then becomes vehemently opposed by Congress, with one of the genuinely nasty Executive-Legislative fights in American history over it taking place under Teddy Roosevelt.
So at the time - and indeed pretty much since the adoption of the Homestead Act - there was a massive industry funneling nominal purchasers to consolidators like mine operators who would pay people (often over and over, given lax record keeping requirements - Union veterans were particularly sought after since they didn't have a 5 year waiting requirement to gain title) to grab chunks of land and mineral rights. This happened all over the place, but the two most applicable to the formation of the Bureau of Investigation were in Oregon and Colorado.
The Oregon scandals were a long series of fraud, with a couple of Congressmen (Binger Hermann most notably) and Senator John Mitchell assisting it over a couple of decades. Mitchell was only one of 12 sitting senators ever to be criminally indicted and only one of 5 convicted; he got six months in prison and died there from an infection after having a tooth removed, which struck many of his friends in Congress as patently unjust punishment - hence one of the reasons for the nasty political fight.
But the 1907 event in Hesperus, Colorado was far worse. It involved the lead Secret Service agent in the region, Joe Walker - who'd accomplished something like 1400 indictments from land fraud over his career, which gives an idea of just how common this was - along with another agent, Thomas Callaghan, and a couple of government contractors who were working as investigators for the Interior Department, John Chapson and Tom Harper.
They went to a homestead claim to investigate a report that it had actually been made on behalf of the Porter Fuel Company, a large coal miner in the region, hitched their horses in a place ominously referred to by locals as "Dead Man's Gulch", found the shaft and immediately realized that given it was reinforced it was an air shaft rather than a well, left Walker above considering he was significantly older than the 3 others and in no shape to rappel down it, went down, explored a bit to confirm it was a mine, tried to come back up, and discovered it'd been sealed off with railroad ties and dirt.
Harper excavated enough to have part of the roof cave in on him, prompting a fall that broke a few ribs with only the logs reinforcing the shaft slowing him enough so that it wasn't fatal, went back up to continue digging, made a hole big enough to exit and tie off a rope they'd brought down, and the other two followed him to the surface.
A short distance away lay the body of Walker, who'd been blasted in the back at very close range with a shotgun (he had at least 12 pellet holes in his back); his revolver was still holstered. They split up to try to better the odds that one would survive to summon help; Chapson stayed behind with the revolver to guard the body, Harper went to a nearby farmhouse he knew had a telephone, and Callaghan went back to town to get the sheriff.
On the way there, he ran into two men on a buggy, one a miner, the other holding a shotgun and identifying himself as Joseph Vanderweide, both the superintendent of Porter Fuel and the nominal owner of the homestead. They told Callaghan they were out with a shotgun on Sunday 'hunting rabbits' (which you don't do with a shotgun if you want to use their meat later); Callaghan arrested them, ran into the sheriff along the way (there had been an 'anonymous tip' of a shooting), and they were brought to the Durango jail.
Vanderweide eventually confessed, but claimed self defense, which was extraordinarily dubious given his victim was shot in the back and that Walker's revolver was untouched. The trial, conducted in state court with a judge and jury of locals opposed to government restriction on land use (for instance, the judge had reduced the 1400 indictments of the grand jury down to one, murder), was a farce and both men were acquitted. The only value of it was that in the investigation, the full plan was revealed: they were going to drop dynamite down the shaft along with Walker's body and claim a gas pocket had exploded, conveniently eliminating any evidence of the murder of all four.
The case made it all the way up to the Supreme Court when the two were indicted on a separate count of conspiracy to commit murder, but the justices ruled the new charge was essentially double jeopardy and the two walked away free. Walker is now generally regarded as the first Secret Service agent murdered in the line of duty.
While the lack of punishment led to outrage in the press, on the other hand Congress was not particularly happy that the Secret Service had not only blown past its statutory authority but was investigating multiple members of it for land fraud. To give you an idea of how upset Congress was, it took the extraordinary step of expunging a message from the President on the subject from its records - the first time that had been done since the Polk administration during the Mexican American War.
With Congress not willing to allow any sort of formal consolidated federal investigative branch, Roosevelt stealthily created what became the Bureau of Investigations in the summer of 1908 with an order issued by his Attorney General and the transfer of 9 Secret Service agents to become the core of a group of 30 investigators who report directly to one of the AG's aides. (Importantly, Congress doesn't find out about this action until the fall when it's essentially a fait accompli.) Their powers and the amount of personnel employed expanded greatly with World War I, largely by default as there was no other agency - including the Marshal Service - capable of the massive amount of record keeping involved in investigating a significant amount of both aliens and citizens.
Finally, keep in mind that the other problem here was very little federal criminal law existed for the nascent Bureau to enforce; Congress had enacted almost none of the legislation we take for granted today and what there was dealt mostly with fraud. In fact, the first modern bit of that type of legislation was the Mann Act in 1910; for a couple of decades it was about the only thing on the books that the Bureau could enforce for violent crime that crossed state borders. Up until the 1930s the Bureau's arrest powers were often unused for the most part; it left almost all criminal cases to local and state police because there really wasn't all that much they could do on the federal level. For that matter, once they finally gained authority, the lawyers and accountants Hoover had hired as agents in the 1920s initially proved a terrible match for the gangland warfare the FBI got thrown into in the 1930s; very few carried firearms up until that point (doing so was urged by FDR's attorney general Homer Cummings rather than Hoover himself), and some had never even fired a weapon before they were tossed into the battles that created the modern image of the agency.
Teddy Roosevelt talks about the Colorado case a little in his autobiography and it's covered a bit in some of the FBI histories, but the best narration of the story and a lot of the context I discuss comes from The Birth of the FBI by Willard Oliver; it's a fascinating read especially if you're curious about the history of law enforcement. I also recommend Beverly Gage's brand new biography released a few days ago on Hoover, G-Man: J. Edgar Hoover and the Making of the American Century; it's likely to be the reference biography on him going forward.