Senior Lead QA Engineer (Agentic AI & Automation)

About the Role:

Grade Level (for internal use):

11

The Team:
Join the elite Quality Engineering arm of S&P Global’s AI transformation. Our team is responsible for the "Trust Layer" of our Agentic Workflows. We don't just test code; we validate AI reasoning. We ensure that our autonomous agents—powered by GCP (Gemini), AWS (Bedrock), and Azure AI Foundry—act reliably and securely within the Moveworks ecosystem.

The Impact:
As a Senior Lead QA Engineer, you are the guardian of AI reliability. You will design the frameworks that test the unpredictable nature of LLMs, ensuring that agentic "thoughts" translate into safe, accurate business actions. Your work prevents hallucinations, secures enterprise data, and ensures a flawless experience for every global employee.

Responsibilities

• Agentic Logic Validation: Design and execute test strategies for autonomous AI agents. You will move beyond functional testing to validate reasoning pathways, tool-use accuracy, and multi-step planning using frameworks like DeepEval or G-Eval.
• Multicloud AI Benchmarking: Oversee the quality of LLM deployments across a multicloud stack. You will benchmark the accuracy and latency of Gemini (GCP), Claude (AWS), and GPT (Azure) to ensure the right model is chosen for each automation.
• Automation-of-Automation Testing: Build automated test suites for Moveworks SparkActions and hybrid low-code/pro-code connectors. Ensure that a change in a ServiceNow or ADO API doesn’t break the AI agent’s ability to execute.
• Evaluation & Observability: Implement LLM evaluation frameworks (e.g., RAGAS, Arize Phoenix, or LangSmith) to monitor agentic "hit rates" and detect drift or regressions in AI performance over time.
• Adversarial & Safety Testing: Lead "Red-Teaming" efforts for AI agents. Test for prompt injections, data leakage, and boundary-crossing to ensure agents never perform unauthorized actions.
• Technical QA Leadership: Lead a pod of QA engineers in Islamabad. Define the standards for AI testing, conduct architectural reviews for testability, and mentor staff on the shift from deterministic to probabilistic testing.
• Stakeholder Trust: Translate technical "quality metrics" into business confidence. Explain to HR or Finance leaders how we have validated that the AI will handle their complex data with 100% reliability.

What We’re Looking For: REQUIRED SKILLS/QUALIFICATIONS

Technical Expertise:

• 8+ years of experience in Quality Engineering, SDET, or Systems Integration Testing. We value candidates who have led the QA for complex business processes on platforms like ServiceNow, Workato, or Salesforce.
• Automation Strategy: 5+ years focused on automated testing of workflows. You should have a proven track record of testing "end-to-end" user journeys across disparate business systems.
• AI & LLM Testing: A strong interest or experience in Testing Generative AI. You understand the challenges of testing LLM-driven decision-making (e.g., handling non-deterministic outputs).
• The "Pro-Code" Core: Expert-level Python (mandatory) for building custom test frameworks, simulating API responses, and automating AI evaluation pipelines.
• Connector & API Testing: Deep experience testing enterprise APIs (REST/SOAP) and handling the authentication/logic bridges between agents and systems like ADO, Workday, or Salesforce.
• Platform Mastery: Experience with Moveworks or similar conversational AI platforms, and familiarity with cloud environments (GCP, AWS, or Azure AI Foundry) for hosting test runners.

Leadership & Soft Skills:

• Test Leadership: Experience leading QA for high-stakes technical projects. You enjoy mentoring others and defining the "Quality Bar" for new technologies.
• Analytical Communication: Exceptional ability to take a complex AI failure and explain the "root cause" (whether it’s the prompt, the model, or the underlying API) to technical and non-technical stakeholders.

Nice to Have:

• Certifications: ISTQB Advanced Level, Google/AWS/Azure Cloud Certifications.
• Agentic Frameworks: Exposure to testing LangChain or LangGraph-based applications.
• Data Science Lite: Basic understanding of vector databases and RAG (Retrieval-Augmented Generation) to better test knowledge-retrieval accuracy.
• Performance Engineering: Experience testing the latency of LLM tokens and API response times.

Why Join Us?

• Test the Untestable: You will be a pioneer in the field of AI Quality Engineering, building the rules for how autonomous agents are validated.
• Multicloud Exposure: Work across the world's leading AI clouds (Google, AWS, Azure).
• High-Impact Growth: At S&P Global, the QA team is a key partner in the development lifecycle, not an afterthought.

Build the "Trust Layer" for agentic automation. Join S&P Global.

What’s In It For You?

Our Mission:

Advancing Essential Intelligence.

Our People:

We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. Join us and help create the critical insights that truly make a difference.

Our Values:
Integrity, Discovery, Partnership

Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals.

Benefits:

We take care of you, so you can take care of business. We care about our people. That’s why we provide everything you—and your career—need to thrive at S&P Global.

Our benefits include: 

• Health & Wellness: Health care coverage designed for the mind and body.

• Flexible Downtime: Generous time off helps keep you energized for your time on.

• Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.

• Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.

• Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families.

• Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.

For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries

Global Hiring and Opportunity at S&P Global:

At S&P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets.

Recruitment Fraud Alert:

If you receive an email from a spglobalind.com domain or any other regionally based domains, it is a scam and should be reported to reportfraud@spglobal.com. S&P Global never requires any candidate to pay money for job applications, interviews, offer letters, “pre-employment training” or for equipment/delivery of equipment. Stay informed and protect yourself from recruitment fraud by reviewing our guidelines, fraudulent domains, and how to report suspicious activity here.

-----------------------------------------------------------

Equal Opportunity Employer

S&P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law.  Only electronic job submissions will be considered for employment.  

 

If you need an accommodation during the application process due to a disability, please send an email to: EEO.Compliance@spglobal.com and your request will be forwarded to the appropriate person.  
 
US Candidates Only: Know Your Rights: Workplace discrimination is illegal

-----------------------------------------------------------

IFTECH202.2 - Middle Professional Tier II (EEO Job Group)