Skywise Engineer

Job Description:

Education: Engineer (or equivalent) degree in computer engineering or computer science. Experience: 6-8 Years of Experience

Job Description

• Develop and maintain data pipelines for efficient data extraction, transformation, and loading (ETL/ELT) processes, utilizing PySpark for distributed big data processing and Polars/Rust for maximal performance and memory safety in single-node bottlenecks.

• Design, implement, and own internal process improvements: automating manual processes, optimizing data delivery latency using high-speed language components, and re-designing infrastructure for greater scalability and cost efficiency.

• Work on the data pipeline operations to operate, maintain, and evolve our decoding pipelines, proposing improvements to automatize and industrialize all processes and ways of working with a focus on data quality and platform stability.

• Support the ramp-up and installations of data pipeline for future airline/aircraft deployment.

• Integrate high-performance Rust-compiled routines (e.g., UDFs) into Python and PySpark workflows to resolve critical performance issues.

Technical Skills

Core High-Performance & Distributed Computing

• Rust: Strong proficiency in Rust for developing memory-safe, highly concurrent, and low-latency data processing microservices or core pipeline components. Familiarity with the Cargo package manager.

• Polars (Expert Level): Mastery of Polars for high-speed, multi-threaded data manipulation on single machines. Deep understanding of the Lazy API, Apache Arrow columnar format, and query optimization techniques (e.g., predicate and projection pushdown).

• Python: Deep expertise in writing production-grade, modular, and reusable code (including Python packaging/wheels). Proven ability to orchestrate complex workflows and tooling.

• PySpark (Expert Level): Mastery of PySpark internals, including:

  • Advanced Performance Tuning: Expertise in diagnosing and resolving bottlenecks using the Spark UI. Deep understanding of Adaptive Query Execution (AQE), data skew mitigation, and optimizing shuffles.

  • Transactional Data: Experience with Delta Lake, Apache Hudi, or Apache Iceberg for building reliable, ACID-compliant Data Lakehouse architectures (handling UPSERTs, Time Travel).

• SQL (Expert Level): Expert proficiency in analytical SQL (window functions, CTEs) and database optimization (indexing, partitioning, query plan analysis).

Data Orchestration & Platform Ownership

• Workflow Orchestration (e.g., Apache Airflow, Dagster): Proven experience in designing, deploying, and maintaining complex, dependency-driven DAGs in production environments, including failure recovery and alerting.

• Infrastructure-as-Code (IaC): Hands-on experience with Terraform or CloudFormation to provision, manage, and secure cloud data resources and compute clusters.

• Cloud Platforms (AWS/Azure/GCP): Hands-on experience with core cloud data services and cost-management practices (e.g., EMR/Dataproc, S3/ADLS/GCS, and serverless compute).

Data Governance & Software Engineering Practices

• Data Quality & Lineage: Experience implementing robust data quality checks (e.g., Great Expectations) and integrating with Data Catalog/Lineage tools.

• Security & Access Control: Practical experience implementing least-privilege access (IAM/RBAC), data encryption, and data masking/tokenization for sensitive data (PII/PHI).

• CI/CD & Testing: Strong knowledge in building automated test, lint, and deployment pipelines (Jenkins, GitLab CI) for data services. Ability to write comprehensive Unit Tests and Data Validation Tests.

• Domain Specific: Palantir (Skywise) experience good to have.

Soft Skills

• Good communication skills, including the ability to articulate the technical trade-offs between distributed (PySpark) and high-speed single-node (Polars/Rust) solutions to technical and non-technical stakeholders.

• Sound knowledge in Agile/SAFe methodologies.

• Ownership Mentality: Ability to handle a subject from demand management, to development, and long-term production support.

• Ability to solve complex networking, data, and software issues.

• Good to have exposure in aviation industries.

This job requires an awareness of any potential compliance risks and a commitment to act with integrity, as the foundation for the Company’s success, reputation and sustainable growth.

Company:

Airbus India Private Limited

Employment Type:

Permanent

-------

Experience Level:

Professional

Job Family:

Digital

By submitting your CV or application you are consenting to Airbus using and storing information about you for monitoring purposes relating to your application or future employment. This information will only be used by Airbus.
Airbus is committed to achieving workforce diversity and creating an inclusive working environment. We welcome all applications irrespective of social and cultural background, age, gender, disability, sexual orientation or religious belief.

Airbus is, and always has been, committed to equal opportunities for all. As such, we will never ask for any type of monetary exchange in the frame of a recruitment process. Any impersonation of Airbus to do so should be reported to emsom@airbus.com.

At Airbus, we support you to work, connect and collaborate more easily and flexibly. Wherever possible, we foster flexible working arrangements to stimulate innovative thinking.