Senior Software Engineer, Observability

NVIDIA's Observability team is seeking a Senior/Staff Engineer to compose and build the next-generation, multi-region observability platform. This platform powers our rapidly expanding AI, Data, and Observability ecosystem, operating at an immense scale: trillions of metrics, hundreds of terabytes of logs, and billions of distributed traces are processed daily across high-performance datacenters and multi-cloud environments. This is a high-impact, architecture-heavy, and code-first role. You will be responsible for architecting, building, and operating NVIDIA’s unified observability stack, which encompasses metrics, logs, traces, profiles, and analytics powered by advanced technology. Your ownership will span the entire telemetry pipeline, including ingestion, storage, query routing, governance, multi-tenant isolation, GPU-accelerated analytics, and real-time insights.

You will play a crucial role in crafting NVIDIA's global observability strategy by collaborating with teams with varied strengths including GPU Compute, Distributed Systems, Networking, ML Infra, AI Platform, and Cloud Services to ensure engineers have deep access to system health, performance, and debugging signals.

What you’ll be doing:

• Designing and operating scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments

• Architecting end-to-end observability pipelines, including ingestion, storage, querying, and visualization

• Extending monitoring and alerting with Prometheus, Alertmanager, Thanos/Mimir, Grafana, and OpenTelemetry

• Building scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Opensearch stacks

• Implementing distributed tracing platforms (Tempo, Jaeger, OpenTelemetry) and integrating with service meshes, load balancers, and APIs

• Defining and driving adoption of SLOs, SLIs, and error budgets across services and teams

• Automating provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python)

• Ensuring reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure)

• Embedding security guidelines into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls

• Mentoring engineers and shaping Nvidia observability strategy and technical roadmap

What we need to see:

• 8+ years of experience with distributed systems, with a focus on observability and monitoring systems

• BS or MS in EE, ECE, CS, or equivalent experience.

• Deep expertise with metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Opensearch), and tracing platforms (Jaeger, Tempo, OpenTelemetry)

• Strong programming skills in Go or Python for automation, operators, and custom integrations

• Experience running observability platforms on Kubernetes and operating them at scale across multi-datacenter environments

• Demonstrated skill in crafting, optimizing, and scaling telemetry pipelines handling high cardinality and high efficiency data

• Solid understanding of distributed systems, performance engineering, and debugging complex workloads

• Familiarity with service meshes, networking, and workload instrumentation (Envoy, Istio, OpenTelemetry SDKs)

• Collaboration skills and the ability to influence engineering teams to adopt observability guidelines

Ways to stand out from the crowd:

• Strong background in software design and system architecture with excellent coding skills in Go and Python for building telemetry pipelines and microservices.

• Experience operating large-scale, multi-region observability platforms (metrics, logs, traces).

• Proven track record to meet strict SLAs for high-throughput, high-cardinality distributed systems.

• Deep expertise in data platforms and telemetry ingestion, storage, and querying layers.

• Demonstrated success collaborating across teams to drive platform improvements and standardization.

NVIDIA leads the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing, and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions, from artificial intelligence to autonomous cars. NVIDIA is looking for exceptional people like you to help us accelerate the next wave of artificial intelligence.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.Applications for this job will be accepted at least until December 12, 2025.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.