Staff Software Engineer - Production Engineering

As a Staff Software Engineer within RBC Borealis' Lumina Production Engineering team, you will serve as a technical leader responsible for developing and maintaining the critical infrastructure that powers RBC's next-generation AI and innovation platforms. You'll work at the intersection of software engineering and operations, ensuring our cutting-edge systems remain reliable, performant, and capable of supporting RBC's digital transformation initiatives.

Production Engineers at RBC Borealis build the foundational systems that enable every major innovation initiative within Lumina's portfolio. Working alongside industry-leading engineers within RBC's innovation hub, you'll contribute to code and systems that directly power breakthrough AI capabilities, advanced analytics platforms, and next-generation customer experiences.

What will you do?

Production Infrastructure Ownership

• Own and operate backend services that power Lumina's AI/ML platforms, real-time analytics engines, and experimental customer-facing applications
• Operate the infrastructure components that drive RBC Borealis' advances in artificial intelligence, machine learning, and data science initiatives
• Manage core services including large-scale data processing pipelines, model serving infrastructure, feature stores, and high-throughput API gateways
• Partner with innovation teams to ensure seamless integration of experimental technologies into RBC's enterprise architecture
   

Technical Leadership & Engineering Excellence

• Lead engineering initiatives by example, mentoring team members and driving technical excellence across Borealis innovation projects
• Write, review, and optimize high-performance code that operates at enterprise scale within fast-moving innovation cycles
• Develop comprehensive documentation, capacity planning models, and operational runbooks for rapidly evolving systems
• Debug complex production issues live on cutting-edge AI/ML infrastructure and experimental platforms
   

Operational Excellence & Incident Response

• Participate in on-call rotations and serve as an escalation contact for critical service incidents affecting Lumina's innovation platforms
• Drive post-incident reviews and implement systematic improvements to prevent recurrence in rapidly evolving systems
• Partner with SRE teams to establish and maintain service level objectives (SLOs) for experimental and production AI workloads
• Champion proactive monitoring, alerting, and automated remediation strategies for novel technology stacks
   

Cross-Functional Collaboration

• Work closely with data scientists, ML engineers, and product teams to ensure reliable deployment and operation of AI/ML models and features
• Collaborate with enterprise security, compliance, and risk teams to maintain regulatory requirements while enabling rapid innovation
• Partner with Borealis platform teams to optimize infrastructure for AI workloads and experimentation velocity
• Interface with external technology partners and vendors supporting RBC's innovation initiatives
   

What do you need to succeed?

Must have:

• 8+ years of experience in production engineering, platform engineering, or similar roles supporting large-scale distributed systems or AI/ML platforms
• Proficiency in at least one of: Python, Go, or Java, with demonstrated experience building production services that support data-intensive workloads
• Extensive experience with container orchestration platforms (Kubernetes, OpenShift) and cloud-native architectures, particularly for AI/ML workloads
• Deep understanding of both analytical and transactional data stores from a data engineering platform perspective (e.g., PostgreSQL, MongoDB, Elasticsearch, Kafka, Redis, vector databases)
• Strong experience with CI/CD pipelines, infrastructure-as-code (Terraform, Ansible), and GitOps workflows supporting rapid iteration cycles
• Proven track record of operating mission-critical systems with high availability requirements in dynamic, fast-paced environments
• Experience with modern observability tools (Prometheus, Grafana, ELK stack) and distributed tracing for complex data pipelines
• Solid understanding of networking fundamentals, load balancing, and content delivery networks supporting high-throughput applications
• Bachelor's degree in Computer Science, Engineering, or equivalent practical experience

Preferred Qualifications:

• Experience supporting AI/ML infrastructure, model serving platforms, or data science workloads in production
• Knowledge of enterprise governance, compliance frameworks, and regulatory requirements in innovation contexts
• Familiarity with modern identity and access management protocols (OAuth2, JWT, mTLS) and zero-trust architectures
• Experience with event-driven architectures, streaming data platforms, and real-time analytics systems
• Understanding of capacity planning and performance optimization for GPU workloads and high-throughput data processing
• Previous experience with incident management processes and on-call responsibilities in rapidly evolving technical environments
• Experience in financial services, fintech, or other innovation-focused technology organizations
• Advanced degree in Computer Science, Data Engineering, or related technical field
   

What makes you stand out:

• Track record of leading technical initiatives that improved reliability and performance of AI/ML or data-intensive systems
• Experience mentoring engineers in fast-paced innovation environments and contributing to technical community growth
• Demonstrated ability to balance technical debt reduction with rapid feature delivery and experimentation
• Strong communication skills with ability to influence both technical and business stakeholders in an innovation context
• Passion for automation, eliminating manual operational overhead, and enabling developer productivity in AI/ML workflows
   

Note: This role requires participation in on-call rotations to ensure 24/7 operational support for critical Lumina systems and innovation platforms. Candidates should be comfortable with incident response responsibilities and working in a fast-paced production environment supporting cutting-edge technology initiatives.

About RBC Borealis

RBC Borealis is the driving force behind Royal Bank of Canada’s AI and data innovation. As part of Canada’s largest financial institution, we bring together a team of architects, engineers, scientists, and product experts on a mission to revolutionize finance through world-class research, solutions, and a resilient data platform. With locations across Toronto, Waterloo, Montreal, Calgary, and Vancouver, we’re at the forefront of AI research and platform development. With a focus on cutting-edge research in areas like time series forecasting, causal machine learning, and responsible AI, we are seamlessly integrating AI research and data engineering, to solve critical challenges in the financial industry. We are building intelligent, and scalable, data-driven solutions that will help communities thrive and drive innovation for our customers across the bank.