St. Victor is unique among the other villages of the Tave River Valley in that it alone hosts a high rocky promontory from which the remains of an old fortress reign. The lords of Sabran built the citadel known as the « Castellas » in approximately 1125. The fort became the defensive center of the area, the Sabranenque, that the counts of Sabran controlled, and St. Victor became the area’s capital. During troubled periods throughout the 12th century, the population found shelter on the northern flank of the tall hill protected by the fortress. Their homes are part of the today’s « old village ».

At the beginning of the 13th century, Count Rostand de Sabran linked his fortunes with those of the Cathar leader Raymond de Toulouse to counter the French King Louis VIII, but they were defeated. Rostand’s power dwindled with the defeat, and he was forced to seek the King’s pardon and swear an oath of allegiance. In exchange he conceded the fortress to the King’s steward in nearby Beaucaire who destroyed the fortifications. St. Victor was reduced to being one fiefdom among others. Over time it was ruled by several of the local reigning families such as the Montlors, the Poitiers, the Nicolais, and the Gadagnes.

During the religious wars of the second half of the 16th century, royal troops were based at the citadel, and then it was abandoned. It has recently been acquired by the village that undertook its restoration, a project expected to take a number of years. The work is being done by a local group, the friends of the Sabranenque. Already the old village shows remarkable efforts of renovation.

The original fort as well as its ramparts were built with limestone from local quarries some of which had been utilized since Roman times. The rounded appearance of the citadel is only an optical illusion as its walls are flat but arranged to achieve the best defensive positions.

The old village had four defensive towers :

•  to the east, the Ormeau tower
• the Gardette or guard tower (also called the Poterne or back gate) that controlled the only entrance to the village; its door hinges and slit which though arrows could be shot are still visible
• the Middle tower
• The West tower that served as an outlook position

The rampart has been conserved along its entirety, about 300 meters. Several parts of the adjacent path are still passable along the western part.  

A second wall exists at the base of the fortress. In this thick wall, one can see sandstone opening for the crossbows; the sandstone comes from local quarries.

The castle keep or internal tower (« donjon ») is the central part of the citadel. A massive fortified entry housed the mechanism that raised or lowered a heavy iron gate.

The interior courtyard is 1500 square meters. Near the east entry, there is a large vaulted room with half moon arches and the bottom part of spiral staircase constructed from local Tavian sandstone that probably led to living areas on the floor above. The chapel is 8 x 5 meters square and has a double, off-centered Romanesque arch. It has a watch tower that rises above the chapel itself; the west face of the tower is made of limestone. It appears that the chapel was the original stucture prior to the castle keep.