Senior Data Engineer

 

Job summary

We’re looking for a Senior Data Engineer who can own modern data platforms end-to-end and help enable AI-powered capabilities across our products. You’ll design and operate reliable, scalable data pipelines on Databricks and collaborate with product and engineering teams to integrate intelligent data-driven solutions. This role is primarily focused on Data Engineering, with opportunities to explore and apply Generative AI and Machine Learning technologies responsibly at scale.

 

What you will do

• Design, build, and optimize batch and streaming pipelines on Databricks (Spark, Delta Lake) for high-volume, mission-critical data.
• Implement robust data modeling, transformation, quality, and metadata practices (expectations, profiling, lineage).
• Ensure reliability and performance of data services with CI/CD, orchestration (e.g., Databricks Workflows/Airflow), and infrastructure-as-code.
• Build observability (logging, metrics, dashboards, alerting) for data and downstream AI services.
• Partner with security, platform, and product teams to strengthen data governance, access control, and cost optimization.
• Collaborate with engineers to deliver LLM- and AI-backed features using OpenAI Agent Builder / Agent Bricks (or similar) and OpenAI/Azure OpenAI APIs.
• Contribute to retrieval pipelines, vector store integrations, and model evaluation processes.
• Participate in prompt design, safety/guardrails, and performance evaluation for applied AI solutions.

 

What you will have

• 5+ years in Data Engineering (or adjacent ML/Data roles) building production-grade data pipelines and platforms.
• Strong proficiency in Python, SQL, and PySpark; deep experience with Databricks and cloud data stacks (AWS or equivalent).
• Expertise in Delta Lake/S3-class storage, version control (Git), and CI/CD for data services.
• Experience building monitoring and dashboards for data or AI services (Grafana or similar).
• Exposure to AI/ML applications in production environments, including LLM or retrieval-augmented workflows.

 

Nice to have

• Hands-on experience with OpenAI Agent Builder / AgentBricks or comparable AI agent frameworks.
• Familiarity with MLflow for experiment and model lifecycle management.
• Working knowledge of LangChain, LlamaIndex, and vector databases.
• Understanding of LLM observability, evaluations, and feedback loops.
• Familiarity with security and governance domains (PKI, identity, data privacy).

 

How you work

• You focus on reliability, scalability, and measurable outcomes.
• You collaborate across product, platform, and security teams to deliver robust, production-ready data and AI solutions.
• You stay curious about new data and AI technologies and apply them thoughtfully to improve business outcomes.

 

Benefits

• Generous time off policies
• Top shelf benefits
• Education, wellness and lifestyle support

 

#LI-KK1