aglan Castle is not part of the original Norman Conquest of Wales and its construction, as it is seen today, was begun by Sir William ap Thomas during the fifteenth century and much of it still remains. Located between Abergavenny and Monmouth in south Wales, the castle was not designed to guard a river crossing or important road. It sits on the site of an earlier manor house.

William ap Thomas married Elizabeth Bloet, the heiress of the Raglan estate. William had fought at Agincourt beside King Henry V and was knighted by Henry VI. He was known as the 'blue knight of Gwent' due to the colour of his armour. When Elizabeth died in 1420 William was able to maintain possession of the castle by an agreement with his stepson. He died in 1445 and his eldest son William Herbert carried on the building work. William Herbert was a Yorkist supporter and was created first Earl of Pembroke. He was executed after defeat at the battle of Edgecote in 1469. His son, William Herbert, the second Earl of Pembroke inherited the castle. This William was married to Mary Woodville the sister of Queen Elizabeth, Edward IV's wife.

The castle has a large six-sided tower known as the 'Yellow Tower of Ghent' due to the colour of its stone. The tower is surrounded by its own moat and accessed via a drawbridge.