NatWest Group’s Head of Responsible AI & AI Strategy, Dr Paul Dongha, outlines how we’re helping to set standards for AI regulatory learnings as part of our work with the FCA.
Why are we collaborating with the FCA?
We’re working with the FCA to strengthen our evidence-led and structured approach to trial one of our agentic AI use cases. This collaboration helps achieve alignment between the bank and the FCA on key risks, builds shared confidence in AI systems, and shapes regulatory learnings and best practice. Our testing is conducted in close collaboration with colleagues, to ensure agentic AI uplifts and augments existing manual processes.
What AI use case are we testing?
We’re testing agentic AI technology to accelerate our complaints handling process by investigating and analysing customer cases. The agentic AI model investigates complaints via multiple data sources and presents a summarised view to a colleague complaint handler for approval – helping accelerate complaint handling and resolution. We ensure that all AI generated summaries are subject to strict human oversight and rigorous controls.
The testing takes place through the FCA’s AI Live Testing. The aim of AI Live Testing is to provide us with a safe place to trial agentic AI systems in real-world conditions, with appropriate regulatory support and oversight.
What does the testing look like in practice?
Testing combines real data with checks at development and model testing phases. During development, we run many tests, while ensuring our agentic AI system operates effectively.
We evaluate our use case model's performance through both human and automated methods, considering metrics like task accuracy, coherence, and hallucination (when AI generates incorrect or misleading information as truth). Performance is tracked daily and reviewed regularly.
What’s next for the collaboration?
We’ll be using production-like environments and monitoring data to get a fuller picture of the potential benefit of our agentic AI system as we look to move from testing through to a live environment.
Our ongoing engagement with the FCA, and lessons learned through testing, will be important in helping to further inform our approach to governance, AI ethics, AI management and operational resilience.
