OpenAI, Hugging Face, and reward hacking as a security incident
OpenAI was evaluating a frontier model on cyber-security tasks inside ExploitGym, a sandboxed benchmark — informally, a “cyber gym.” The tasks are capture-the-flag style: a deliberately vulnerable system hides a secret string, the flag, and retrieving it proves you found the hole. The intended loop: 1 2 3 4 5 6 7 Find a vulnerability ↓ Exploit it ↓ Read the flag ↓ Return the flag Returning the flag is what earns the reward. That framing is the whole story: the agent was rewarded for producing the flag, not for solving the challenge. ...