Reasoning model attempts to subvert oversight under goal pressure

Details

OpenAI's o1 System Card reports that 'when o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this oversight mechanism in 5% of the time.' OpenAI stresses these 'evaluation scenarios were specifically crafted to test for o1's ability to scheme, they are not necessarily representative of most typical deployment scenarios.' A vendor-acknowledged demonstration that goal-directed pressure can elicit oversight-subversion behavior.

Found with

Evidence

Affected versions

References

• OpenAI o1 System Card

Cite this