The Town Pond was once located south of Richmond Road just east of Arthur Kill Road. Its existence is documented in the early 18th century, when Richmond was established as the county seat. The pond was likely drained and filled with soil around the 1820s, allowing construction of the Richmond County Hall which was completed ca. 1829. Richmond County Hall was demolished in 1890. A parish house for St. Andrew's Church stood on the site from about 1891 to 1929.
In the 1960s this area was the focus of an archaeological investigation; artifacts found there are now in the collections at Historic Richmond Town.