Chisholm wanted nothing to do with secession, and though he had two sons who fought with the 15th Virginia Infantry Regiment during the war, Confederate officers passing through Hanover Courthouse threatened to hang him for his anti-secessionist tendencies.
Working in the Tavern's kitchens during Chisholm's time was Martha Ann Fields, a slave of the Winstons from nearby Nutshell
. Fields escaped in the confusion following a battle in July of 1863, and managed to take six of her children to freedom at Fort Monroe
in Hampton, Virginia. Her son George Washington Fields went on to become the first African-American to graduate from Cornell University in 1890, and his brother James became a lawyer and then a Virginia State Senator in 1889.
Chisholm's politics did him no good whatsoever when the Yankees came to town. They helped themselves to whatever took their fancy, ultimately appropriating around $3,000 worth of the Tavern's resources.