IN_Senior Associate_GCP,BigQuesry,GCS,DataFlow_Data and Analytics_Advisory_Bangalore

Job description

We are seeking a talented and experienced GCP Data Engineer to design, build, and maintain scalable and reliable data pipelines and infrastructure on the Google Cloud Platform. In this role, you will be crucial in transforming raw data into actionable insights, enabling data-driven decision-making across the organization by working closely with data scientists, analysts, and other stakeholders.

· Design, develop, and maintain high performance ETL/ELT pipelines using Pyspark, Python, SQL

· Build & optimize distributed data processing workflows on cloud platforms (GCP or Azure)

· Develop and maintain batch and real-time ingestion including integration with Kafka

· Ensure Data Quality and metadata management across data pipelines

· Monitor and tune data systems and queries for optimal performance, cost-efficiency, and reliability.

· Automate data workflows and processes using tools like Cloud Composer (Apache Airflow) and leverage Cloud Monitoring/Logging for troubleshooting and operational efficiency.

Data engineering with 4-8 years of experience with strong proficiency in PySpark, Python, SQL.

· Hands-on experience with GCP especially on the services like BigQuery, DataProc, Cloud Storage, Composer, Dataflow

· Strong understanding of data warehousing concepts, data modelling & ETL/ELT processes and expertise in Datawarehouse / Datalake / lakehouse architecture

· Familiarity with big data processing frameworks like Apache Spark and should have experience in Apache Kafka

· Experience with version control tools like Git and CI/CD pipelines.

Good to have:

· Experiences with DBT – in building models, testing & deployments

· Should have knowledge on Data modelling

· Good to have exposure on Docker and deployments on GCP

· Good to have hands-on with Pub/sub, Cloud run

· Exposure to streaming workloads

· Good to have hands-on exposure Java – core

· Analytical and problem-solving skills

· Ability to work in agile

· Communication and stakeholder management skills

· Accountability & ownership

DE - Cortex

Data engineer with hands-on expertise on Google Cloud Cortex Framework focusing on data integration, analytics, and AI/ML solutions using SAP data on Google Cloud Platform (GCP).

Responsibilities

· Design, build, and deploy enterprise-grade data solutions that bridge SAP and Google Cloud environments using the Cortex Framework's reference architecture and deployment accelerators.

· Utilize tools like SAP SLT Replication Server, the BigQuery Connector for SAP, and Dataflow pipelines to ingest, transform, and load (ETL/ELT) SAP data into BigQuery.

· Leverage predefined data models, operational data marts in BigQuery, and tools like Looker to create dashboards and deliver actionable business insights from unified SAP and non-SAP data.

· Implement machine learning templates and integrate AI models (potentially using Vertex AI) to optimize business outcomes and enable advanced analytics.

· Work with business stakeholders, engineering teams, and partners to ensure solutions meet business needs and are scalable, cost-effective, and compliant with best practices.

· Monitor system performance, troubleshoot data pipeline issues, and implement best practices for data governance and security within the GCP environment.

Mandatory Skill Sets

· Should have knowledge on Data modelling

· Good to have exposure on Docker and deployments on GCP

· Good to have hands-on with Pub/sub, Cloud run

· Exposure to streaming workloads

· Good to have hands-on exposure Java – core

Preferred Skill sets:

· Strong hands-on experience with Google Cloud Platform (GCP) services, especially BigQuery, GCS, Dataflow, Cloud Composer and Vertex AI.

· Proficiency in SAP systems (SAP ECC or S/4HANA) and SAP data extraction methods.

· Expertise in SQL and Python programming languages.

· Hands-on expertise on Looker Studio (LookML)

· Familiarity with data governance, data modeling, and security principles.

· Specific knowledge of the Google Cloud Cortex Framework for SAP integration and analytics is a mandatory skill.

· Experience deploying Cortex Framework components from the official GitHub repository.

Years of Experience required:4Years to 8Years

Education Qualification :BE,B.Tech,MCA.