William de Albini (or Aubigny) became the owner of the castle and the first Earl of Arundel either having been granted it by Henry I or by the fact that he married Henry's widow Adeliza of Louvain, sources disagree. In the dispute between Matilda and King Stephen, William de Albini gave refuge to Matilda at the castle. When Matilda, Henry's daughter, arrived in England she sought the assistance of Adeliza her step-mother and Adeliza had agreed. Arundel Castle was besieged by King Stephen in reponse but the castle did not fall to the King and Matilda escaped. Several generations of Albinis owned Arundel Castle until Hugh, the last male heir, died. The castle became the property of John Fitzalan who married Hugh's daughter Isabel in 1243. At the end of the thirteenth century Richard Fitzalan began a phase of reconstruction work that included the barbican that adds protection to the Norman gatehouse. The Fitzalan family remained almost continually in posession of the castle until in 1555 Lady Mary Fitzalan married Thomas Howard, the 4th Duke of Norfolk, and the Howard family became the owners of the castle.
During the English Civil War the castle was held by Royalist forces. The Parliamentarians under the command of William Waller besieged the castle from late 1643 until the spring of 1644. Cannons were used on the castle and did much damage. Over the centuries the castle's structure was changed and repaired so that the castle we see now is not an original Norman castle. But there are sections of the keep and gatehouse that retain earlier Norman work.