Principal Engineer – Gen AI Platform Inferencing Engineering

About this role:

Wells Fargo is seeking a Principal Engineer – Gen AI Platform Inferencing Engineering to lead the development and optimization of our AI model serving and inferencing platforms within Digital Technology's AI Capability Engineering group.

This is a software engineering role — you'll write code, build systems, and solve hard problems in the AI inference stack. You'll work deep inside frameworks like vLLM, SGLang, and NVIDIA Dynamo, extending and optimizing them to serve models at enterprise scale. You'll also build the automation, tooling, and deployment infrastructure that connects these runtimes to Kubernetes-native serving layers like KServe, KNative, and OpenShift AI.

If you've contributed to inference frameworks, written custom serving logic, or built production ML serving pipelines in Python, we want to hear from you.

In this role, you will:

• Develop, extend, and optimize inference runtime configurations and integrations across vLLM, SGLang, NVIDIA Dynamo, TensorRT-LLM, and Triton
• Write Python-based tooling and automation for model onboarding, serving configuration, performance benchmarking, and deployment pipelines
• Build and maintain Kubernetes-native model serving infrastructure using KServe, KNative, and OpenShift AI — including custom serving runtimes and inference graphs
• Implement and tune inference performance optimizations — continuous batching, speculative decoding, prefix caching, concurrency control, autoscaling policies, and disaggregated prefill/decode pipelines
• Develop Helm charts, operators, and Kustomize overlays for deploying and managing inference workloads on OpenShift/OCP
• Integrate inference platforms with GPU workload orchestrators (Run:AI or similar) — automating project provisioning, quota management, and workload scheduling
• Build observability and testing harnesses — load testing frameworks, latency/throughput profiling scripts, and regression test suites for inference stack upgrades
• Partner with AI/ML teams to productionize new models, defining serving architectures, resource requirements, and SLA targets

Required Qualifications:

• 7+ years in software engineering or platform engineering (work experience, training, military experience, or education)
• 5+ years of programming experience in Python with experience building production systems

Desired Qualifications:

• Experience with Inference frameworks, such as vLLM, SGLang, NVIDIA Dynamo, TensorRT-LLM, or Triton Inference Server
• Experience with Kubernetes-native ML serving, such as KServe, KNative, Seldon, or OpenShift AI
• Experience with Inference optimization, (Continuous batching, speculative decoding, KV-cache management, prefix caching, quantization-aware serving (FP8, AWQ, GPTQ), or tensor parallelism configuration)
• Experience with Container platform development, (Writing Helm charts, operators, or custom controllers for OpenShift, GKE, or EKS)
• Experience with GPU workload orchestration, (Run:AI, Kueue, Volcano — scripting workload automation, quota management, or scheduler integrations)
• Experience with Performance and load testing, (Building benchmarking tools for token throughput, time-to-first-token, batch latency, and autoscaling behavior)
• Familiarity with NVIDIA GPU fundamentals (CUDA, MIG, NCCL), experience contributing to open-source inference projects, or background in ML observability tooling (Prometheus, Grafana, Arize)

Job Expectations:

• This position is not eligible for Visa sponsorship
• This position requires a hybrid in office work schedule

Pay Range
 

Reflected is the base pay range offered for this position. Pay may vary depending on factors including but not limited to demonstrated examples of prior performance, skills, experience, or work location. Employees may also be eligible for incentive opportunities.

$159,000.00 - $305,000.00

Posting End Date:

28 Apr 2026

*Job posting may come down early due to volume of applicants.

We Value Equal Opportunity

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.

Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.

Applicants with Disabilities

To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo.

Drug and Alcohol Policy

 

Wells Fargo maintains a drug free workplace.  Please see our Drug and Alcohol Policy to learn more.

Wells Fargo Recruitment and Hiring Requirements:

a. Third-Party recordings are prohibited unless authorized by Wells Fargo.

b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.