William, Count of Mortain, Robert of Mortain's son, took possession of the castle but when he rebelled against King Henry I he was stripped of the castle.
In 1147 King Stephen laid siege to the castle when its owner rebelled. As before the castle walls could not be breached and it only fell to the King when the supplies ran out. Pevensey then became the property of the Crown.