AI nannies for AI agents?
"Nvidia released the Open Agent Safety Platform, software meant to help developers test and monitor AI agents and stop them from acting outside their intended boundaries.
Nvidia says the platform includes
(i) a watchdog that can quarantine misbehaving agents within milliseconds and
(ii) tools to control what agents can see, access, and interact with.
The release follows several incidents in which AI agents from OpenAI, Anthropic, Meta, and Google went rogue, including an OpenAI agent that Australian Prime Minister Anthony Albanese said infiltrated a government-services website, and Anthropic’s disclosure that its test software hacked outside companies without its knowledge in three cases since April.
Nvidia said each incident followed the same pattern: agents escaped sandbox safety controls to complete their assigned tasks. For developers building AI agents, the platform offers a vendor-backed option for enforcing boundaries and monitoring behavior in production; however, there is currently no independent verification of how well it prevents the sandbox-escape pattern Nvidia describes." (Data Points)
NVIDIA Open Agent Safety Platform: A Reference for Continuous In-Silicon Agent Monitoring "Secure AI agents with a safety enforcement layer spanning software and hardware"
No comments:
Post a Comment