Construction of the castle is thought to have started in the mid twelfth century after the site was granted by King Stephen. It is mentioned in the royal accounts of Henry II when payments for repairs were made. The first castle on the site would have been a motte and bailey type fortification primarily made of wood consisting of an inner ward with a square tower surrounded by a moat and a small cresent-shaped bailey. The keep was demolished in the middle of the 13th Century and several large towers were added to the curtain wall of the inner bailey. The entrance to the inner bailey was then protected by two large round towers. A larger bailey was then created on the opposite side of the inner bailey from the original smaller bailey. This larger bailey was walled and had several towers protecting its perimeter.
In 1201, during the reign of King John, Hubert de Burgh was granted many castles in the area including Grosmont, Skenfrith and White Castles. But his ownership of the castles did not last long as the castles were granted to King John's favourite William de Braose in 1205. In 1210 de Braose fell out of favour with the King and fled to France where he died in the following year. In 1219 By 1219 Hubert de Burgh was Justicier and Earl of Kent and regained ownership of all three castles. But again, like de Braose, Hubert fell from favour and in 1232 the castles were granted to Peter de Rivaux. White Castle eventually became the property of the Crown and was owned by Prince Edward, the future King Edward I, and then to his brother Edmund, Crouchback, the Earl of Lancaster. With the defeat of Llewelyn in 1282 the military importance of White Castle came to an end.