Job Description
Data Engineer
Responsibilities
Data Platform Development & Implementation:
Design, develop, and implement scalable and efficient data pipelines using GCP services (e.g., Dataflow, Cloud Functions, Workflows) for ETL/ELT processes, adhering to established architectural guidelines.
Build robust and scalable solutions for orchestrating data workflows, handling dependencies, and ensuring data integrity.
Develop and optimize data models and data warehouse solutions in BigQuery, ensuring performance and data integrity.
Write complex and highly optimized SQL queries to extract, transform, and load data from various sources
Define and enforce best practices, standards, and guidelines for data platform development and usage.
Evaluate new technologies and methodologies to continuously improve our data platform capabilities and efficiency.
Technical contribution & Collaboration:
Provide technical expertise and guidance to junior team members.
Participate actively in code reviews and contribute to improving code quality and standards.
Collaborate closely with the Data Platform Tech Lead, data scientists, analysts, and other stakeholders to understand data requirements and propose effective technical solutions.
Contribute to the documentation of data pipelines, data models, and development processes.
Proactively identify and troubleshoot complex data-related issues and implement effective solutions.
Stay up-to-date with the latest data engineering trends and technologies and share knowledge with the team.
Consulting and Collaboration
Consulting various data delivery teams related to data onboarding and other data platform capabilities
Consult with data analysts on data modeling, product design, and BI tool utilization.
Collaborate with business stakeholders to understand their data needs and translate them into technical requirements.
Platform Administration and Management:
Administer and manage the business intelligence platform Sisense and understanding its capabilities
Ensure the security and compliance of the data platform, implementing appropriate measures to protect sensitive information
.
Requirements
Technical Skills:
Strong proficiency in Python and SQL, including advanced SQL query writing and optimization. Java is also good to have.
Experience with Google Cloud Platform (GCP) services, such as BigQuery, Cloud Run, Data Catalog, Cloud Functions, IAM, GCS, Monitoring, Workflows, Cloud SQL, and Secret Manager.
Solid understanding of backend development concepts, including CI/CD pipelines (CircleCI), Docker, and microservices architecture.
Knowledge of data modeling, data architecture, data pipelines, ETL/ELT processes, and business intelligence tools.
Familiarity with business intelligence tools, with experience building dashboards in Sisense being a significant advantage.
Exp. in shell scripting.
AWS is good to have.
Soft Skills:
Excellent communication and interpersonal skills to effectively collaborate with team members and stakeholders.
Strong problem-solving and analytical abilities to identify and resolve complex technical challenges.
Ability to work independently and manage tasks effectively.
A strong passion for data and a commitment to delivering high-quality data solutions
Additional Considerations:
Experience with data mesh principles and practices would be a significant advantage.
Familiarity with data governance and compliance frameworks is desirable.