A Challenge Answered: The Battle of Kelly’s Ford, March 17, 1863
William Woods Averell was usually considered an even-tempered individual. However, in the opening weeks of March, 1863 his blood had been brought to a boil. This mild mannered New Yorker, whose great grandfather had signed the Declaration of Independence, had his heart set on revenge.
Averell was a drugstore clerk before he received his appointment to the U.S. Military Academy at West Point in 1851. His closest friend at the Academy was from one of Virginia’s oldest and most distinguished families, Fitzhugh Lee. When he graduated, Averell received a commission in the Regiment of Mounted Rifles (later re-christened the 3d U.S. Cavalry). While serving in New Mexico in 1859, Averell was seriously wounded fighting Indians. This wound would put him out of the service for two years. By the time he was fully healed and ready to return to the saddle, North and South were squaring off in bitter conflict. Despite the long recuperation, Averell was made the Acting Assistant Adjutant General on the staff of General Andrew Porter. It was in this position that he witnessed the Battle of First Manassas in July, 1861. A little over a month later, Averell received the Colonelcy of the 3d Pennsylvania Cavalry.
Averell would command a brigade during the Peninsula Campaign and at Fredericksburg. When Major General Joseph Hooker took command of the Army of the Potomac early in 1863 he consolidated the mounted forces into one corps. Under this reorganization, Averell received command of a division. Averell’s chief assignment that winter was to set up a cordon of pickets to protect the Union Army’s winter encampment.
This dull duty was shattered on February 25. In what has become known as the Battle of Hartwood Church, a contingent of Confederate cavalry attacked and drove in Averell’s pickets in an afternoon of spirited fighting. The Rebels were led by Averell’s old friend from West Point, Fitz Lee. Before breaking off the fight, Lee left behind a surgeon to care for his wounded. The doctor also carried a note for Averell.
Please let this surgeon assist in taking care of my wounded. I ride a pretty fast horse, but I think yours can beat mine. I wish you’d quit your shooting and get out of my State and go home. If you won’t go home, why don’t you come pay me a visit. Send me over a bag of coffee. Good-bye, Fitz”.
This gamely jab from Lee added to Averell’s embarrassment for the sudden surprise. The affair at Hartwood Church also grabbed the attention of General Hooker. When he called on Averell a few days after the fight, Averell requested permission to return the favor upon the Rebels. Hooker responded that in the not too distant future, Averell would have his chance.
Averell would chomp at the bit for about two weeks, waiting for his opportunity. Then, orders came from Hooker instructing him to move up the Rappahannock and attack the Confederates at Culpeper Court House. It was there that recent intelligence placed Lee’s brigade.
On the morning of March 16, the column moved out. Altogether, Averell would have around 2,100 men to put into the field against any opposition. He had along his own brigades, consisting of three regiments a piece, commanded by Alfred Duffie and John McIntosh. Duffie headed the 4th New York, 6th Ohio and 1st Rhode Island. McIntosh led three regiments from the Keystone State, the 3d, 4th and 16th Pennsylvania Cavalry. Averell also could rely on the services of the 1st U.S. Cavalry and three squadrons from the 5th U.S. Cavalry, commanded by Captain Marcus A. Reno. The entire force was complemented by the 6th New York Light Independent Battery.
After a long day’s march, the column bedded down in the hamlet of Morrisville. Averell’s scouts continued onward to Mount Holly Church. They would report back signs of a Confederate camp in the direction of Rappahannock Station (the modern town of Remington) just east of Culpeper. Instead of continuing on to Rappahannock Station the next morning, Averell decided to turn directly toward the Rappahannock from his present position and cross the river at Kelly’s Ford. At 4 a.m. on the morning of March 17, the Yankees saddled up and rode toward the ford.
The Rebels had taken the necessary precautions to guard Kelly’s Ford. Elements from the 2d and 4th Virginia Cavalry were positioned there to oppose any attempted maneuvers. They did indeed give Averell a warm reception when the assaults began at sunrise the next morning. The Virginians were able to hold their own for awhile, until troopers from the 1st Rhode Island and 6th Ohio were finally able to force a crossing and gain possession of the south bank. The Confederates withdrew in the face of the Federal onslaught. Knowing that it would not be long until a larger enemy force appeared on the scene, Averell got his division across the river as quickly as he could, arrayed them in formation and began a leisurely advance toward Culpeper.
Word of the Yankees at Kelly’s Ford was not long in reaching Fitzhugh Lee at Culpeper. Curiously, Lee’s services in the ante-bellum years and through the early stages of the war were similar to his friend Averell’s. He had been wounded on the frontier while fighting Indians served as a staff officer at First Manassas. Like Averell, he would receive the command of a single regiment of cavalry in the late summer of 1861. Lee would receive a promotion to Brigadier General for his services on the Peninsula. When news came of the Federal crossing, Lee immediately set out with his entire brigade, consisting of the 1st, 2d, 3d, 4th and 5th Virginia Cavalry Regiments. Also accompanying Lee were Cavalry Corps commander Major General J.E.B Stuart and Stuart’s Chief of Artillery, Major John Pelham. For the upcoming battle, Stuart would act merely in the capacity of an observer and leave command of all tactical decisions to Lee. Arriving on the field, the Confederates were greeted by the sight of Averell’s advancing division. For the next several hours, Lee and Averell would engage in a see-saw chess match across the open fields northwest of Kelly’s Ford.
Lee was the first to initiate the action, sending a dismounted squadron forward to probe the Union lines. Averell countered by dismounting and advancing the 4th New York and 4th Pennsylvania along his left. John McIntosh sent the 16th Pennsylvania to extend the line to the right in an attempt to squeeze off Lee’s troopers, who had taken up position behind a stonewall. The maneuver was successful. To counter Averell’s attacks, Lee sent the 3d and 5th Virginia forward in a mounted charge. Joining in with the Virginians was Major Pelham. As he rode forward, Pelham would be struck by a piece of shrapnel that entered the back of the skull. Carried from the field, the young Alabaman would pass away quietly the next morning in Culpeper.
The Federals stoutly held their ground in the face of the assault and repulsed the attacking Confederates. In an effort to follow up their success, Colonel Alfred Duffie led the 1st Rhode Island, elements of the 4th Pennsylvania and the 6th Ohio forward. Two squadrons of Reno’s 5th U.S. Cavalry joined in and a saber swinging melee ensued. To meet Duffie, Lee personally led the 1st, 2d and 4th Virginia forward in a counterattack. The 1st Rhode Island would rode ahead to engage the Virginians. Supported again by elements of the 6th Ohio and 5th U.S., the Union troopers once again held their own. With the attack blunted, Lee decided to pull back and regroup.
With his regiments scattered, Averell also paused to reform. After getting his men back in position, Averell advanced his division. With the Federals still not willing to give up the field, Lee planned another attack. He decided to send the 1st, 3d and 5th Regiments against Averell’s right, while the 2d and 4th assaulted the Union left. Once again, the Yankees held firm. Carbine fire from dismounted troopers repulsed the attack on Averell’s right. Over on the left, rather than waiting for the Rebel charge to hit their lines, the 1st Rhode Island, 6th Ohio and elements of Reno’s Regulars went out to meet their counterparts. The two lines collided and the Federals gained the upper hand, forcing the Virginians to retreat.
Surprisingly, rather than pursue the Confederates, Averell elected to break off the action and withdraw from the field. Darkness was setting in and the erroneous reports of the arrival of Confederate infantry in the area contributed to the decision. A duty fitting to their stature, Averell deployed Reno’s Regulars as a rear guard and began his withdrawal. The Confederates followed at a distance. While covering the retreat, Captain Reno would have his horse shot out from under him. The mount would fall on top of him, giving Reno a hernia. For his actions that day, George Custer’s future subordinate would receive a brevet to the rank of Major.
As Averell neared the ford, he remembered the slight Fitz Lee had left for him just a few weeks previous. In a house near the river, Averell left behind a surgeon with two wounded officers. These men held a bag of coffee and a reply to Lee. It read:
Here’s your coffee. Here’s your visit. How do you like it? How’s that horse? Averell.
After the crossing, the troopers made their way to Morrisville. The next morning they began their march back to their Stafford camps, arriving there late that afternoon.
The friendly squabble between two old friends begun at Hartwood Church would play witness to a transformation at Kelly’s Ford. For the first time in the Eastern Theater, a large force of Union cavalry had ventured into enemy territory and engaged the Confederates. Derisively criticized for their lack of aggressiveness and fighting prowess in the past, Averell’s troopers landed blow after blow against Lee. The Yankees more than proved they were capable of standing toe to toe with the Rebels. This valuable combat experience would play dividends in the months to come, on fields in Virginia and Pennsylvania.
Averell would continue in command of his division through the Chancellorsville Campaign. Unfortunately, he would become a scapegoat for Hooker’s defeat and would be relieved of command and transferred from the Army of the Potomac. Back in command of a division during the Valley Campaign of 1864, he would be blamed for the failure to cut off Jubal Early after Fisher’s Hill and would be relieved again. Despite running afoul of two army commanders, Averell would receive brevets to the rank of Brigadier and Major General before the end of the war. A month after Lee’s surrender at Appomattox, Averell resigned from the U.S. Army. The following year, Averell received the post of consul general to British North Africa and would hold the post until 1869. He would become a successful inventor toward the end of the nineteenth century. Averell died in Bath, New York on February 3, 1900.
Although elevated to division command, Fitz Lee would be passed up to command the Cavalry Corps upon Stuart’s death the following spring. Lee would serve faithfully under his new chief and would be wounded at Third Winchester. It was not until Hampton’s transfer in January 1865, that Lee would be given corps command. He would serve in this position through the waning days of the war. With the end of hostilities, Lee tried his hand as a farmer in Stafford County. In 1885, he would successfully run for Virginia’s governorship. After a failed U.S. Senate bid in 1893, President Cleveland appointed Lee as consul in Cuba. When war broke out with Spain, Lee would be commissioned a Major General of U.S. Volunteers. Retiring from the service in 1901, Lee passed away four years later and is buried in Hollywood Cemetery in Richmond.
One can wonder that in their twilight years, if Lee and Averell thought back to their friendly rivalry in a time of war. This not so ordinary rivalry had an incredible impact on the cavalries of both sides. It was a new beginning wrought on the banks of the upper Rappahannock and a St. Patrick’s Day to remember.
All images courtesy of the Library of Congress.
For more information on the Battle of Kelly’s Ford see The Union Cavalry Comes of Age: Hartwood Church to Brandy Station, 1863 by Eric J. Wittenberg. Brassey’s Incorporated, 2003.
4 Responses to A Challenge Answered: The Battle of Kelly’s Ford, March 17, 1863
Just in case these men ever become numbers in a page of statistics . . . It is these remembrances that make them whole again.