History is Generated by the Algorithms

“Who is telling the story?” is one of the central mantras of the history profession. Finding out what sources of information are being used is crucial to creating a full, fair, and critical evaluation of it. In the age of AI, however, a new question must increasingly be considered: “Who is generating the story?” Although the text of generative AI might seem like original work at first glance, it draws heavily from pre-existing sources of information. Consequently, determining precisely what these sources are is crucial for evaluating the accuracy and utility of any product of artificial intelligence.

Due to it being embedded into Google, Gemini is one of the most widely used AI models. When a search is entered into Google’s search engine, the Gemini AI Overview is frequently one of, if not the, top result on the page. This prominent positioning gives its output an outsized impact in shaping how users gain knowledge and answer questions. Fortunately for the purposes of this research, the AI Overview provides the main sources it used to generate its responses, typically between five to fifteen. It should be noted that it is not an exhaustive list of all references, but it is still useful in gauging what is primarily shaping the output.

To see what sources Gemini uses to explain the history of the American Civil War, I decided to run an experiment. First, I gathered a list of Civil War-related queries to search into Google’s search engine. I used the forty-five battles designated as decisive by the 1993 Report on the Nation’s Civil War Battlefields as well as the fifty prominent leaders depicted on the 1885 Kurz & Allison print Prominent Union and Confederate Generals and Statesmen as They Appeared During the Great Civil War, 1861-5.

I then turned off personalized settings on Google to create as neutral a result as possible. With that ready, I began to work my way through the list, searching each item and recording the sources listed by the AI Overview. Several hours and 18,042 spreadsheet cells later, I had my data.

The top ten most frequently cited sources, with the percentage of overviews they appeared in were:

1. Wikipedia (97%)
2. American Battlefield Trust (80%)
3. YouTube (65%)
4. National Park Service (56%)
5. Encyclopedia Britannica (41%)
6. History.com (36%)
7. Encyclopedia Virginia (22%)
8. Facebook (22%)
9. American History Central (17%)
10. EBSCO (16%)

To borrow the title of a great Civil War movie, these results demonstrate the good, the bad, and the ugly of the Geminification of Civil War history. Some strong, reliable sources do make the list. The American Battlefield Trust, the National Park Service, Encyclopedia Virginia, and EBSCO (a database of academic articles) are all welcome sights on the list. Encyclopedia Britannica, History.com, and American History Central are a decided step down in quality, but still are passable sources of information, especially for a beginner looking for basic facts. That the near ubiquitous Wikipedia takes the top spot is sure to raise eyebrows, depending on what stance one takes on the open source encyclopedia. The most questionable results, however, are YouTube and Facebook.

In fairness to both sites, there is valuable and rigorous history on both. For instance, the most frequently cited YouTube channels (History Gone Wilder, Warhawk, and Life on the Civil War Research Trail, in that order) all belong in that category in the opinion of the author. Despite these diamonds in the rough, relying on social media for accurate history is questionable at best. Especially worthy of note is that in many cases, especially with Facebook, the cited material appeared to also be AI generated. For those who rightly assess human historical research to be better than that of artificial intelligence, this is certainly a concerning development.

Other interesting data points emerge as well. This site, for instance, was the twenty-seventh most cited, appearing in three percent of searches. The dearly departed HistoryNet does better, coming in thirteenth and appearing in twelve percent of searches. PBS, meanwhile, places eleventh with a fifteen percent appearance rate. Reddit’s appearance in six percent of searches places it higher than the Smithsonian’s three percent, Library of Congress’s two percent, and National Archives’s two percent. That most prestigious and venerable of Civil War publications, USA Today, appeared as a source in one overview.

Going by category, the most frequently cited type of site were non-academic history sites. Admittedly, this is a broad categorization, including everything from Emerging Civil War and the American Battlefield Trust to HistoryNet and the Confederation of the Union Generals. Unsurprisingly, the quality of these sites can vary. While the four aforementioned sites are widely known and respected among Civil War circles as sources of information, random sites drawn from the depths of the Internet can be more suspect in their information.

Government-owned websites (such as the Library of Congress, National Archives, and Congressional bioguide) are the second most cited, followed by academic sources. For the latter, this refers more to the websites of universities and their presses than actual academic publications.

This gets at perhaps the largest problem of the Gemini AI Overview as a source of history. While it can cite quality websites, and even an academic article on occasion, books are entirely absent from its sources (at least those that it lists). This is not to say that books are the be-all and end-all of historical information, but it is not a contested contention that they form an important part of it. They can get into more depth of detail and complexity than most webpages typically feature. It could be argued that neither is necessary for something explicitly labeled as an “overview.” Nevertheless, it does seem wrong to exclude the long-time foundation of humanity’s knowledge from contributing to the new frontier of information.

What can be said of AI generated history? Others in this series provide commentary on its accuracy, so I leave the determination of that factor to them. In regards to its sourcing, however, AI is neither fully commendable nor condemnable. Its frequently questionable sources, circular citations of other AI-produced material, and total exclusion of books as sources is a black mark that should not be overlooked. It does also frequently cite reliable websites as sources, but overall, it is certainly better to follow the links to those websites for information rather than using the overview for your Civil War facts.

Part of a series.