The first castle on the site was a timber Motte and Bailey constructed at the end of the 11th century (Approx 1081 when the Normans came to Wales). The original stone building was built in around 1150 and subsequently modified, the tower being added inside existing stone walls at a later date as extra fortification. The Court (manor house building) was started by John Picard in around 1300 but the existing house was built in approximately 1450 by Roger Vaughan. Roger Vaughan was known as the wealthiest 'commoner' in Wales. During the War of the Roses Roger Vaughan changed sides but managed to end up on the Yorkist side, for his help in the war he was Knighted. By the early 15th century, the castle had probably been partially abondoned in favour of the new house. The Court remained the residence of the Vaughan family until 1783 when the building passed out of their hands and became a farm. The vast majority of the paranormal sightings are of a Lady dressed in white with green eyes, mostly in Bedroom 2 opposite the Main Bed Chamber, and she is said to be anxious but not aggressive. Along the Gallery and Battlements a woman in white is also seen pacing up and down and anxiously looking out over as if looking for someone to return. In the Courtroom a little boy has been seen sitting by the door.