Vocera, now part of Stryker, is seeking a visionary and hands-on Principal Engineer – AI Test, Evaluation & Data Architecture to define and lead the enterprise-wide strategy for AI validation, model evaluation, and data governance across our speech and GenAI platforms.
This role serves as the AI Quality Architect for real-time speech systems, NLP pipelines, and LLM-powered applications deployed in mission-critical healthcare environments. You will establish scalable evaluation frameworks, design AI testing platforms, define data governance standards, and ensure production reliability of AI systems at scale.
This is a high-impact architectural leadership role requiring deep expertise in LLM validation, RAG evaluation, speech benchmarking, automation, MLOps, and AI lifecycle governance.
Define and own the end-to-end AI evaluation architecture across speech, NLP, and GenAI platforms.
Establish standardized evaluation frameworks for:
ASR systems (WER, latency, robustness, domain adaptation),
NLP systems (intent accuracy, entity F1, confusion analysis),
LLM systems (hallucination rate, groundedness, factual accuracy, consistency, safety)
Define measurable AI quality SLAs and release gating criteria.
Architect benchmarking standards across model versions, prompt changes, and retrieval updates.
Institutionalize regression evaluation pipelines for all AI releases.
Architect validation frameworks for:
RAG-based systems,
Prompt orchestration workflows,
Multi-agent or multi-model AI pipelines
Define groundedness measurement strategies for enterprise RAG.
Establish adversarial testing, stress testing, and edge-case validation frameworks.
Implement hallucination detection standards and mitigation measurement.
Drive responsible AI practices, including bias detection and safety validation.
Design and lead implementation of a scalable AI testing platform that includes:
Offline evaluation pipelines,
Golden dataset-driven regression systems,
Synthetic data generation frameworks,
Online A/B testing & shadow deployment strategies
Integrate AI validation workflows into CI/CD and MLOps pipelines.
Define drift detection and performance degradation monitoring strategies.
Establish real-time observability dashboards for AI quality metrics.
Define enterprise-wide data governance strategy for AI systems, including:
Data collection and curation standards,
Annotation workflows and validation,
Dataset versioning and reproducibility,
Traceability across model iterations
Establish gold datasets for:
Speech systems,
NLP pipelines,
Clinical and conversational workflows
Drive continuous learning loops between production telemetry and training data.
Ensure compliance with healthcare data privacy and regulatory standards.
Define evaluation strategies for:
Accent variability,
Noisy clinical environments,
Domain-specific vocabulary adaptation
Establish measurable latency and reliability benchmarks for real-time AI systems.
Lead failure mode analysis and systemic AI quality improvements.
Serve as the principal authority on AI testing and evaluation strategy.
Influence architecture decisions alongside Principal AI Architects and platform leaders.
Mentor senior engineers in AI validation, benchmarking, and data governance practices.
Drive AI quality maturity across multiple pods and engineering teams.
Partner with Product and Executive stakeholders to align AI quality metrics with business outcomes.
Shape long-term AI reliability roadmap for the organization.
Bachelor’s or Master’s degree in Computer Science, Engineering, AI, or related field.
13+ years of experience in software engineering, AI engineering, or AI validation roles.
5+ years of hands-on experience with LLM, RAG, NLP, or speech-based AI platforms.
Proven experience designing AI evaluation or testing frameworks at scale.
Strong expertise in:
Hallucination detection,
Golden dataset regression strategies,
Adversarial and edge-case testing,
Prompt validation and benchmarking
Strong proficiency in Python and data analysis for AI evaluation.
Experience building automated AI validation pipelines integrated with CI/CD.
Strong system design and distributed architecture understanding.
Experience leading cross-team technical initiatives.
Experience in architecting evaluation frameworks for production RAG systems.
Familiarity with semantic search validation and retrieval benchmarking.
Experience designing LLM guardrails and structured output validation.
Knowledge of Responsible AI, fairness evaluation, and compliance auditing.
Experience evaluating ASR/TTS systems in production environments.
Strong understanding of speech benchmarking metrics and domain adaptation strategies.
Experience with Azure ML, Azure OpenAI, Azure AI Search.
Familiarity with MLOps and model lifecycle automation.
Experience designing scalable evaluation infrastructure in cloud-native environments.