This may not be the earliest but it's definitely an early example. The case is Marcy vs. Barnes in the Supreme Court of Massachusetts, 82 Mass. 161, October 1860. It's basically a case where one of the parties claimed that his signature on a promissory note was forged. High resolution photographs were taken of the supposed forged signature and an expert in handwriting analysis was brought in to analyze those photos.

It was argued in court that the photographs shouldn't be admitted as evidence. This was the court's decision regarding that:

The magnified photographic copies of the genuine signatures of the defendant, and of the disputed signature, which was submitted to the inspection of the jury, were, we think, in connection with the testimony of Mr. Southworth, admissible in evidence. Assuming it to be true, as he testified, which yet was a fact first to be considered and determined by the jury, that the copies were accurate in all respects, excepting only in relation to size and color, they were capable of affording some aid in comparing and examining the different specimens of handwriting which were exhibited on the trial. It is not dissimilar to the examination with a magnifying-glass. Proportions are so enlarged thereby to the vision, that faint lines and marks, as well as the genuine characteristics of handwriting which perhaps could not otherwise be clearly discerned and appreciated, are thus disclosed to observation, and afford additional and useful means of making comparisons between admitted signatures and one which is alleged to be only an imitation. Under proper precautions in relation to the preliminary proof as to the exactness and accuracy of the copies produced by the art of the photographer, we are unable to perceive any valid objection to the use of such prepared representations of original and genuine signatures as evidence competent to be considered and weighed by the jury.

It's entirely possible that photography was used or attempted to be used in cases prior to this but I haven't come across anything in the literature. There is an article titled The Legal Relations of Photography from the January 25, 1873 issue of The Albany Law Journal, 7 Alb. L.J. 50. It's clear that photography wasn't well understood, by legal scholars at least, at the time as there's mention of a dead person's final view of the world being imprinted on the retina for a while and perhaps accessible via a photograph.

However it mentions two other cases in addition to Marcy. One is The Taylor Will Case, a probate case in the State of New York from 1871 (10 Abb. N.S. 300) where photographic evidence was disallowed because it was considered secondary evidence. There's also Ruloff's Case, also from New York in 1871 that allowed photographs of deceased robbers for identification purposes (45 N.Y. 213).

So while these are later cases, it shows, and the article agrees, that there hadn't really been enough of these in front of courts to come to a decision at the time whether photographic evidence should be allowed or what conditions should be set to allow them.