Civil War Photo Sleuth Uses Facial Recognition Software to ID Portraits

Luther, KurtBy ECW Correspondent Joe Giglio

A stroke of luck at a Civil War exhibit in Pittsburgh in 2013 allowed Kurt Luther to find a photo of his great-great-uncle Oliver Croxton. “It was an incredible moment,” recalls Luther. “First of all, it was so unlikely. Photos of Civil War generals and colonels are pretty easy to come by, but as you move down the ranks, the odds of finding a surviving portrait drop dramatically.”

An infantryman from Pennsylvania’s 134th volunteer regiment, Croxton was a low-ranked soldier of modest means, and the portrait Luther discovered was likely one of the few—if not the only—photographs taken of the man during his life. Even if Croxton had the means or desire to have a multitude of photos taken of himself, there’s no guarantee Luther’s search would have been any easier. “The photos would have to avoid being damaged or lost for 150 years,” he explained. “And even then, only 10-20% of Civil War portraits are identified.”

But this picture of Luther’s ancestor was of remarkable quality. “You couldn’t ask for a better Civil War portrait of an ancestor,” he said. “The photo was in beautiful condition, and the photographer knew what he was doing. You can see details like the fine lines around his eyes and the roughness of his hands. He’s got a great beard. He’s wearing his Union fatigue blouse and corporal’s chevrons and even some unique hat brass showing his regiment and company.”

Luther is well aware of how lucky he was to find his ancestor at all, let alone such a high-quality picture of him. “It’s a big world out there, and a lot of Civil War photos are in private collections,” he said. “This wasn’t the result of an internet search, but simply being in the right place at the right time. If I hadn’t made the trip to that museum and looked in that particular display case, it never would have happened.”

The experience of seeing an actual photo of his ancestor by chance, after searching for him for some time, had a profound effect on Luther. “It’s that intangible sense of getting to know somebody in a way that you can’t from the military records and the biographical information you can dig up,” he explained.

After finally seeing Croxton, Luther was moved to expand his knowledge of Civil War photography and, combined with his experience as a professor of computer science at Virginia Tech, he would launch Civil War Photo Sleuth five years later in August of 2018.

Civil War Photo Sleuth is a free website created to use cutting-edge facial recognition software and user input to identify the tens of thousands of unidentified civil war photos in existence.


According to Luther, the site currently has more than 12,000 registered users who have uploaded more than 7,000 photos. Added to the mix are some 21,000 images from public collections such as the Library of Congress and the National Archives.

Using Microsoft’s Azure Cognitive Services, the site has led to more than 300 identifications by its user base.

While the service is impressive, some of the Microsoft software it uses has had some issues. The technology came under scrutiny in 2018 when an academic study found that it and other facial recognition software preformed noticeably worse on people of color and woman than it did on white men. “This is definitely important for Civil War Photo Sleuth because we have many photos of US Colored Troops, as well as a growing collection of images of Civil War-era women,” says Luther.

Microsoft has made positive strides with the technology, however. In addition to attempting to address the problems found in 2018, the site has increased the number of faces that can be compared in one search. “Originally, we were limited to comparing an unknown face to a collection of up to 1,000 identified faces,” Luther said. “As we collected tens of thousands of photos, we had to create more than 20 lists of 1000 faces each, and the software had to search through all of them, one at a time. Now, Microsoft allows each face list to include up to a million faces. That’s a much more streamlined process, which means faster results for the user.”

Readers can get involved in three main ways, Luther said. “In increasingly level of difficulty, they are 1) search for Civil War photos based on name, location, etc., 2) add identified photos to expand the reference database, and 3) try to identify unknown photos.”

While there are plenty of identifications made by users, this does not mean that all of them are correct. In the first month of the site’s existence, there were 119 identifications of unknown photos, 88 of which were unique people. From there, Luther explained, they have to manually judge all of the identifications. “Since this is original research, we can’t always know for sure,” he explained, “so we allowed the expert to provide a rating from 1 (definitely wrong) to 4 (definitely correct). If we consider ratings of 3 (possibly correct) or 4 (definitely correct) to be ‘positive matches,’ then 75 of the 88 user identifications (about 85%) were positive matches.” While Luther has not brought in an expert since, he is hopeful that the 85% positive match rate will extend to the 200 or so identifications made in the last year.

Luther said that he is interested in all of the identifications but is always particularly excited for identifications of photos in public collections “because the information benefits such a broad audience.”

On August 1, Civil War Photo Sleuth celebrated its one-year anniversary. The site remains free to join. For more information, visit


2 Responses to Civil War Photo Sleuth Uses Facial Recognition Software to ID Portraits

  1. Adding technology into the historical mix is such a positive move. From DNA testing to ground-penetrating radar, we are beginning to cut into the numbers of unknowns. The facial reconstruction on the Hunley sailors brings tears to my eyes, as does their DNA efforts. Digitalizing old images adds life and vibrancy to photographs and makes them more accessible to a younger audience. One of my favorites in this area is a portrait of general Sherman that has been Photoshopped to make him smile. He looks kind, and amused by the process. I’d like to have seen Sherman smile IRL, but this is a stride forward. Huzzah!

Please leave a comment and join the discussion!