Like many houses in Virginia, Forest Hill began its existence as a small vernacular house, but was enlarged to its present size and given Federal-style characteristics such as the graining on the interior woodwork and the fanlight above the front door. The original 1803 section was enlarged in 1816 by then owner William Macon Waller, a prominent county resident, who held the positions of justice of the peace, school commissioner, sheriff, and delegate to the Virginia legislature. Waller, who gave the property its name, amassed approximately 1,200 acres during his lifetime.It was worked by approximately 45 slaves. Waller had 12 children by two wives; the first wife, Elizabeth Mutter, died not long after they moved to Forest Hill, but his second wife, Sarah Armistead Garland, lived on the property after Waller’s death in 1849. After her death, the Waller family retained the property until 1921. The current 200 acres contain numerous dependencies, including 19th-century agricultural buildings, early 20th-century storage buildings, and possibly archaeological remains of a family cemetery and slave quarters. Together with the house, they effectively convey their association with a prominent 19th-century Amherst County landholder and citizen.