Software Engineer, Discovery Eval

((Insert Job Group Template))

((Include reporting structure and work modality ie: In this hybrid role, you will report to a Senior Research Scientist))

You will:

• Design, develop, and implement automated eval analytic workflows using large language models and other AI techniques to analyze and categorize events from perception evaluation systems.
• Build and maintain scalable data pipelines to automatically cluster, summarize, and surface critical issue categories discovered through large-scale simulation and real-world data.
• Develop an orchestration system to manage the lifecycle of discovered issues, including data sampling, annotation, and integration into the ML training and evaluation flywheel.
• Enhance and support core discovery tools and frameworks, focusing on improving the precision, recall, and reliability of automated issue detection.
• Expand the capabilities of discovery evaluation systems to cover a broader range of perception signals and other advanced outputs.
• Collaborate with other engineers, ML infrastructure teams, and researchers to integrate cutting-edge AI techniques and build robust, end-to-end evaluation solutions.

You have:

• BS or MS in Computer Science or a related field.
• 2+ years of software development experience (3+ years for L5).
• Strong programming skills in Python and/or C++.
• Experience with data analysis, data pipelines, and working with large datasets.
• Solid understanding of machine learning concepts and evaluation methodologies.
• Excellent problem-solving skills and ability to work independently and collaboratively.

We prefer:

• Experience in developing or utilizing AI-powered agentic workflows.
• Familiarity with evaluation framework in autonomous vehicle systems or robotics.
• Hands-on experience with ML infrastructure and tools.
• Experience with large-scale data processing systems.
• Proven ability to design and implement novel evaluation metrics and techniques.
• Experience with data visualization tools and techniques.