A bank or credit union evaluates an AI TPRM agent by testing it on vendors whose files staff already know, checking that every finding carries a source and retrieval date, confirming that a person approves each recommendation and that the record survives for examiners.
This article covers the evaluation’s scope, five steps, a scorecard, the guidance and research that apply, and how the Ask Kaia Third-Party Risk Intelligence Agent maps to it.
What an AI TPRM Agent Evaluation Covers
An evaluation has two objects:
- The agent’s output (whether it matches the right legal entity, finds the sanctions, enforcement and insolvency records that exist, shows where each finding came from and routes recommendations to a person)
- The agent’s provider, a third party in its own right, together with the model and data providers the agent depends on.
Third-party AI risk sits in that second object above. The evaluation applies to banks and credit unions considering an agent for due diligence or ongoing monitoring.
General TPRM software selection, with its feature lists and requests for proposal, runs first or in parallel. The institution’s enterprise AI policy sets the governance standard, and the evaluation tests the agent against it.
Evaluating an AI TPRM Agent in Five Steps
1. Define the decisions
List the tasks the agent will perform, such as entity resolution, screening or contract extraction, and the decisions that stay with staff, such as approval, remediation and exit.
2. Test on known vendors
Run the agent on vendors whose due diligence files the team has already completed, including some with known adverse findings, so its output can be checked against an answer key.
3. Compare output with the analyst file
Check entity matches, missed and false findings, and whether every finding shows its source and retrieval date.
4. Test the controls around the agent
Confirm that recommendations wait for human review, that escalations reach the right owner and that the record keeps inputs, outputs and the analyst’s decision.
5. Diligence the provider
Apply the 2023 interagency due diligence factors to the provider (information security, operational resilience, reliance on subcontractors including model providers, and contract terms on data use, audit rights and exit).
AI TPRM Agent Scorecard
Each criterion in this AI agent evaluation pairs with the evidence to request and the source that anchors it, and is rated meets, partly meets or does not meet. The institution sets any weights and pass marks in its own policy.
| Criterion | Evidence to request | Anchor | Rating |
|---|---|---|---|
| Legal-entity resolution | Test results on known vendors, including ambiguous names | 2023 guidance C.2.b (ownership structure) | |
| Source and retrieval date on every finding | Sample profiles showing source links and dates | 2023 guidance D.3 (documentation) | |
| Evidence sufficiency shown | Display of confidence, missing items and stale data | 2023 guidance C.2 (documented limitations) | |
| Screening coverage | List of sanctions, debarment, enforcement and insolvency sources | 2023 guidance C.2.b and C.2.c | |
| Control mapping | Findings mapped to the institution’s controls, with gaps listed | Institution’s control framework | |
| Human approval before action | Workflow showing who approves each recommendation | NIST AI 600-1 (human-AI configuration) | |
| Audit record | Export of inputs, outputs, decisions and limitations | 2023 guidance D.3 | |
| Provider due diligence | Security, resilience and subcontractor disclosures, including model and data providers | 2023 guidance C.2.g, C.2.i, C.2.l; NIST AI 600-1 (value chain) | |
| Exit and data return | Contract terms for termination, data return and access removal | 2023 guidance C.3 and C.5 |
A gap on source dating, human approval or the audit record leaves no way to show an examiner how a conclusion was reached.
What the Guidance and Research Say
The 2023 Interagency Guidance on Third-Party Relationships treats the agent’s provider like any other third party. A banking organization’s use of third parties does not diminish its responsibility, and reliance on subcontractors is one of the 14 due diligence factors.
Model risk management guidance no longer covers these agents: the guidance the Federal Reserve, FDIC and OCC issued on 17 April 2026 (OCC Bulletin 2026-13) places generative and agentic AI models outside its scope. The institution’s AI policy and its third-party program therefore set the standard an agent must meet.
NIST’s Generative AI Profile (AI 600-1, 26 July 2024) names 12 risks, including confabulation and value chain and component integration, which covers non-transparent third-party components and data.
Ask Kaia Third-Party Risk Intelligence Agent
The Ask Kaia Third-Party Risk Intelligence Agent shows source dates, confidence and missing information for each finding and marks information as fresh, aging or stale. Recommendations carry reasons, suggested controls and supporting evidence and require human approval.
The agent maps evidence to the institution’s controls and keeps findings, analyst decisions and documented limitations in a due-diligence record. Each capability lines up with a scorecard row, so a bank can test it against vendors it already knows.
Frequently Asked Questions
Does SR 11-7 Apply to AI Agents?
No. On 17 April 2026 the Federal Reserve, FDIC and OCC replaced the 2011 model risk guidance, which included SR 11-7 and OCC Bulletin 2011-12, and the replacement states that generative AI and agentic AI models are not within its scope. Banks govern AI agents through their own AI policies, third-party risk management and the controls they test before and after adoption.
Is the Provider of an AI TPRM Agent a Third Party?
Yes. The 2023 interagency guidance covers any business arrangement between a banking organization and another entity, and an agent provider that processes vendor data and supports risk decisions fits that definition. Due diligence, contract terms and ongoing monitoring scale to the risk of the activity the agent supports, and the provider’s own model and data providers fall under reliance on subcontractors.
What Is Third-Party AI Risk?
Third-party AI risk is the risk an institution takes on through AI systems that a vendor builds, hosts or embeds in its service, including errors in the output, use of the institution’s data and dependence on the vendor’s own model providers. It applies to AI-specific tools such as TPRM agents and to conventional vendors that add AI features to existing services.
The Predict360 Enterprise Risk Management Software ensures managers have complete visibility of enterprise risk on a single dashboard.
Request Demo- Cloud-Based
- Risk Repository
- Assess Risks
- Real-time Monitoring