I can't say much about specifically colonial America, but I have done some reading in the Ratcliff Highway murders, which were somewhat contemporary, and give some insight into the methodology behind early criminal investigations.
The Ratcliff Highway murders involved two home invasions just outside London in december of 1811. Seven people were killed, and authorities eventually arrested a man named John Williams for the crime, who killed himself before he could be hanged.
The investigation into the murders relied on eyewitness testimony, wound analysis, and physical evidence.
Eyewitness testimony was used to create a basic timeline of events, such as when a boarder living with the Marrs came home to find the Marr family dead, accounts from various night watchmen, and other men who stayed at the boarding house that Williams stayed in. Among the evidence was the claim that Williams had come home late on the night of the second murder and would not let his roommate light a candle and that he was damp smelled of blood, and that the laundress of the boarding house had washed a shirt stained (lightly) with blood.
After the murder of the Marr family, investigators of the River Thames police force examined the scene and determined that a maul, or heavy hammer, had been used to batter the victims to death and the infant son's throat had been slashed open . The second set of murders, at the Williamson home, a crowbar and a sharp knife or razor had been used.
The maul they found at the first scene was chipped on one end and marked with the initials "JP," who was eventually identified as a John Peterson, a sailor who stayed at the same boarding house as John Williams, the Pear Tree. his trunk was in the house, and it was missing the maul. Police connected the two. The owner of the Pear tree even claimed that he had once borrowed the maul and was the one that had chipped it, positively connecting the murder weapon with John Peterson's trunk. Williams, already a suspect due to earlier witness testimony, also had access to the trunk and a bevy of other circumstantial evidence against him, was arrested but never brought to trial, as he killed himself.
You can see a lot of similarities in how this case was investigated and how modern police investigate crimes. Obviously in 1811 things like fingerprint comparison, DNA testing, and other more rigorously scientific processes were impossible, but there was the ability to basically account for ironclad facts. Connecting the maul as the murder weapon and tracking down its owner and tying all the physical evidence to witness testimony helped to make a pretty fair case against Williams.
The case was more or less strong, but largely circumstantial and modern analysis has attempted to debunk him as the perpetrator. That said, the investigation itself is pretty insightful into how pre-20th century criminal investigation was carried out.
Most of this info comes from PD James' The Maul and the Pear Tree. If you're interested in the case I would strongly urge you to track it down and read it - it was easily obtainable in my local library. Sorry I couldn't be more specific about Colonial America, but I hope this helped.
The very top answer of one of the top threads of all time in this sub covers this "What If?" in Massachusetts Bay in 1772; before some rules clarifications happened around here. I highly recommend this thread. It was huge, and what brought me into this sub.
One of my ancestors, John Billington( http://en.m.wikipedia.org/wiki/John_Billington ), was the first person executed in the colonies. Found guilty of murder by a jury and sentenced to hang. It is a very interesting story and a perfect example to answer your question. The colonies did have their own governing bodies and judicial system, albeit with severe oversight from the British gov. Edit: here is a better link that has an explanation of how the trial proceedings took place http://historyofmassachusetts.org/john-billington-mayflower-pilgrim-executed-for-murder/