The model is not the risky part
Why AI risk often comes from access, permissions, tools, memory, and review paths surrounding the model.
TLDR
- The main risk is often not how clever the model is, but what the surrounding system lets it see and do.
- A weak model with broad access can be more dangerous than a strong model inside a narrow reviewed workflow.
- Evaluate data access, memory, tools, permissions, review, and recovery as one operating system.
The title is deliberately incomplete. Models can be wrong, invent facts, miss context, overstate confidence, and reproduce harmful patterns.
But in professional work, risk often becomes consequential through the system around the model.
What can it read? What can it remember? What can it change, send, approve, or trigger? Who checks it, and what happens after a mistake?
Capability is not permission
A capable model with no system access is mostly an adviser. A less capable model connected to confidential files, email, billing records, and external sending can create a much larger operational risk.
That risk comes from composition. A model may receive untrusted content, have access to sensitive information, and possess a tool that can transmit or modify data. OWASP’s guidance for generative AI highlights prompt injection, excessive agency, sensitive-information disclosure, and improper output handling as distinct but interacting risks 2.
Review the surrounding system
For each workflow, inspect:
- data access and purpose limitation;
- identity, role, and permission boundaries;
- what conversation or organisational memory persists;
- tools that can produce external effects;
- approval and escalation paths;
- logs that allow reconstruction;
- reversal, correction, and incident response.
NIST’s AI Risk Management Framework treats trustworthy AI as an organisational and lifecycle practice spanning governance, context, measurement, and management 1. That wider view matters because model evaluation alone cannot tell you whether the deployed workflow is acceptable.
Ask the operational question
“Is the model safe?” is too broad to produce a useful answer.
Ask instead: given these sources, permissions, tools, people, and consequences, what failures are possible, who would notice, and how would the organisation recover?
The model still matters. The operating boundary determines how much its failure can matter.
Sources
/ Start
Start with one business outcome. Expand from there.
Begin with a focused review rhythm, workflow, or team where better operating context would immediately change the quality of preparation and judgment.