Job Title: AI Evaluation Lead
Division: Data, Technology and Innovation
Department: AI Product Delivery
Salary: National (Edinburgh and Leeds) ranging from £72,100 to £100,000 and London from £79,300 to £110,000 per annum (salary offered will be based on skills and experience)
This role is graded as: Technical Specialist – Regulatory
Your external recruitment contact is Benjamin via benjamin.paulon@fca.org.uk.
Your internal recruitment contact is Lauren via Lauren.Pyrah@fca.org.uk
Applications must be submitted through our online portal. Applications sent via social media or email will not be accepted.
About the FCA and team
We regulate financial services firms in the UK, to keep financial markets fair, thriving and effective. By joining us, you’ll play a key part in protecting consumers, driving economic growth, and shaping the future of UK finance services.
The Data, Technology and Innovation (DTI) division enables the FCA to be a digital-first, data-led smart regulator by delivering a secure, agile, and cost-effective technology and data ecosystem that drives better decisions, transparency, and operational efficiency.
Working alongside the wider AI Programme (which will continue to oversee/coordinate AI activity across the FCA), the department will partner with business leads to shape and deliver work in priority areas — Authorisations, SPC, EMO and Anti‑Money Laundering.
Role responsibilities
Define and own evaluation frameworks for GenAI outputs covering quality measures such as accuracy relevance robustness and hallucination rates
Design curate and govern test datasets and benchmarks to ensure consistent model and solution performance assessment over time
Support development of automated evaluation pipelines and operational reporting to embed assurance into delivery
Identify assess and mitigate risks in model behaviour for example bias errors safety concerns and edge cases with clear escalation and control recommendations
Manage delivery through well-defined work packages setting priorities operating standards and performance objectives including line management of Business Analysts
Engage senior stakeholders to understand strategic priorities build a pipeline of scoped and prioritised projects and translate needs into analytics led solutions with clear business value
Work in the public interest protecting 40 million UK consumers who rely on financial services and supporting long term economic growth from an industry contributing 12% of UK economic output
Manage digital and data-led change by encouraging innovative experiments and working with senior stakeholders while empowering a diverse team to collaborate openly
Skills required
Minimum:
Experience delivering analytics, data science and AI/ML initiatives, including defining success measures, evaluating model/product performance, and applying innovative problem-solving approaches
Demonstrated experience leading people (line management, coaching, and performance reviews), capable of overseeing a portfolio of projects and adjusting delivery as priorities shift
Effective stakeholder management skills, including working with senior colleagues to translate priorities into well-scoped, prioritised work with clear outcomes and measurable value
Essential:
Demonstrable experience designing and applying evaluation frameworks for GenAI ML solutions such as accuracy relevance robustness consistency and hallucination or error rates including defining clear acceptance thresholds
Experience curating documenting and governing test datasets and benchmarks including version control to enable repeatable assessment and comparability over time
Ability to identify assess and mitigate model risks including bias safety concerns data leakage harmful outputs and edge cases and to recommend appropriate controls and escalation routes
Experience building or specifying automated evaluation and monitoring approaches such as scripted test runs scoring and dashboards or management information to embed assurance into delivery and ongoing operations
Demonstrated analytical skills with the ability to interpret evaluation results communicate uncertainty and limitations translate complex technical concepts for senior stakeholders and make evidence-based recommendations to improve solution performance
Consistent delivery discipline including planning and managing evaluation work packages prioritising across competing demands and ensuring outputs meet agreed standards and timelines
Clear written documentation skills producing evaluation reports test plans and assurance artefacts that are audit ready and suitable for governance forums
Benefits
28 days annual leave plus bank holidays
Hybrid model where employees work a minimum of 40% in the office each month (expectation of 50% for senior leaders). Changing from September to a minimum of 50% in the office each month (expectation of 60% for Directors and Executive Directors)
Non-contributory pension (8–12% depending on age) and life assurance at eight times your salary
Private healthcare with Bupa, income protection, and 24/7 Employee Assistance
35 hours of paid volunteering annually
A flexible benefits scheme designed around your lifestyle
For a full list of our benefits, and our recruitment process as a whole visit our benefits page.
Our values & culture
Our colleagues are the key to our success as a regulator. We are committed to fostering a diverse and inclusive culture: one that’s free from discrimination and bias, celebrates difference, and supports colleagues to deliver at their best. We believe that our differences and similarities enable us to be a better organisation – one that makes better decisions, drives innovation, and delivers better regulation.
If you require any adjustments due to a disability or condition, your recruiter is here to help - reach out for tailored support.
We welcome diverse working styles and aim to find flexible solutions that suit both the role and individual needs, including options like part-time and job sharing where applicable.
Disability Confident: our hiring approach
We’re proud to be a Disability Confident Employer, and therefore, people or individuals with disabilities and long-term conditions who best meet the minimum criteria for a role will go through to the next stage of the recruitment process. In cases of high application volumes, we may progress applicants whose experience most closely matches the role’s key requirements.
Useful information and timeline
Advert Closing: 19th May
CV Review/Shortlist: 22nd May
First Stage Interviews W/C: 1st June
Second Stage Interviews W/C: 8th June
Your Recruiter will discuss the process in detail with you during screening for the role, therefore, please make them aware if you are going to be unavailable for any date during this time.