Job Summary
Synechron is seeking a highly experienced PySpark Data Engineer to design, develop, and maintain scalable, high-quality data pipelines within the Cloudera Data Platform (CDP). This role is critical in ensuring reliable data ingestion, transformation, and availability for advanced business analytics, reporting, and data science initiatives. The successful candidate will bring a strong background in big data processing, data architecture, and cloud integration, contributing to data-driven decision-making and operational excellence across the organization.
Software Requirements
Required:
Advanced proficiency in PySpark, including handling RDDs, DataFrames, Spark SQL, and optimization techniques
Hands-on experience with Cloudera Data Platform (CDP) components such as Cloudera Manager, Hive, Impala, HDFS, and HBase
Working knowledge of Hadoop ecosystem, Kafka, and distributed data processing tools
Experience with SQL-based data warehousing tools like Hive and Impala
Scripting skills in Linux (Bash, Python) for automation and operational tasks
Familiarity with orchestration and scheduling tools such as Apache Airflow or Oozie
Preferred:
Knowledge of cloud-native data services (AWS Glue, EMR, Azure Data Factory)
Use of version control systems like Git and CI/CD pipelines (Jenkins, GitLab CI)
Experience with data modeling, data governance, and metadata management tools
Overall Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark within the Cloudera Data Platform.
Manage end-to-end data ingestion processes from multiple sources (relational databases, APIs, file systems) into data lakes or warehouses.
Execute data transformation, cleansing, and aggregation processes supporting analytical and reporting requirements.
Conduct performance tuning of Spark jobs and related CDP components to ensure efficient resource utilization.
Implement data validation and quality checks, ensuring data accuracy and consistency through monitoring and alerting.
Automate data workflows using orchestration tools like Airflow or Oozie to reduce manual intervention.
Monitor pipeline performance, troubleshoot failures, and implement corrective actions for operational stability.
Collaborate with data architects, analysts, and data scientists to support large-scale analytics initiatives.
Document data architecture, pipeline configurations, and operational procedures for ongoing maintenance and governance.
Lead data architecture discussions supporting data privacy, security, and compliance standards.
Technical Skills (By Category)
Programming Languages (Essential):
Python (especially PySpark)
SQL for data extraction, validation, and analysis
Big Data & Data Management (Essential):
Spark (PySpark), Hadoop ecosystem, HDFS, Hive, Impala, HBase
Data ingestion and transformation in large distributed environments
Cloud & Platform Technologies (Preferred):
Cloud-native data processing (AWS EMR, Azure HDInsight, GCP Dataproc)
Frameworks & Libraries (Essential):
Spark SQL, Spark Streaming
Data modeling and governance tools (preferred: Apache Atlas or Collibra)
Orchestration & Automation (Preferred):
Airflow, Oozie, Jenkins
Security & Data Governance (Preferred):
Data masking, encryption, access control in distributed systems
Experience Requirements
Minimum of 5+ years as a Data Engineer with deep expertise in PySpark and big data processing
Proven experience designing, implementing, and maintaining scalable data pipelines in enterprise environments
Strong background with Cloudera Data Platform (CDP) components such as Hive, Impala, HDFS, and HBase
Demonstrated ability to optimize Spark jobs and manage high-volume data workflows
Support experience in cloud environments (AWS, Azure, or GCP) for data processing is advantageous
Industry experience supporting financial services, banking, or highly regulated sectors is a plus
Alternative pathways include extensive hands-on Big Data processing experience in data-centric roles with demonstrated expertise in performance tuning and operational stability
Day-to-Day Activities
Develop and optimize Spark (PySpark) data pipelines for ingesting, transforming, and publishing data in large distributed systems
Monitor data workflows and troubleshoot issues proactively to maintain pipeline health.
Collaborate with data scientists, analysts, and platform teams to meet data quality, security, and governance standards.
Automate operational workflows, including job scheduling, alerting, and resource management.
Perform performance tuning of Spark jobs and related components to optimize runtime and resource efficiency.
Conduct data validation, anomaly detection, and data quality assessments.
Document architecture, data flows, and operational procedures for compliance and knowledge sharing.
Support ongoing system upgrades, data privacy initiatives, and cloud migration efforts.
Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or equivalent
5+ years of hands-on experience in data engineering, with an emphasis on PySpark and big data systems
Proven expertise in designing scalable, high-performance data pipelines in enterprise environments
Hands-on experience with Cloudera Data Platform (CDP), Hadoop, Hive, Impala, and HBase
Strong SQL and data modeling skills within distributed data architectures
Experience with cloud data services is a plus
Relevant certifications (e.g., AWS Data Analytics Specialty, GCP Professional Data Engineer) are advantageous
Strong analytical, troubleshooting, and communication skills
Professional Competencies
Critical thinking and analytical mindset for complex data workflows and problem resolution
Ability to manage multiple priorities and deliver results in a fast-paced environment
Effective collaboration skills for cross-team data initiatives and stakeholder engagement
Innovation-driven approach for optimizing and automating data processes
Ownership mindset to ensure operational stability and data quality standards
Adaptability and continuous learner to keep pace with evolving big data and cloud technologies
SYNECHRON’S DIVERSITY & INCLUSION STATEMENT
Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.
All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.