The building was built in 1816 as a private house by Thomas Cooke, however the Cellars are part of a far older house that was built on the site in the 16th Century. The Orme family are known to have owned a house on this site from 1538, while local magistrat Cooke and his wife Charlotte built the existing building and used it as their townhouse from 1816 to 1856. It is alse thought that Thomas Cooke used to hold trials in the kitchen of the house. Thomas and Charlotte divorced amid a lot of scandal after which it was sold to 3rd Earl Fitzwilliam who gave permission for the building to be used as the city's first hospital. It serves as The Peterborough Infirmary from 1857 to 1928, while there was apparently a big fire in the building in 1884. The building was then bought by the Museum Society and became a museum in 1931.