Sr. Data Engineer

We're looking for a highly skilled and experienced Senior Data Engineer to join our team. You'll be instrumental in designing, building, and maintaining our next-generation Data Observability platform and its infrastructure. This role requires a deep understanding of distributed systems and hands-on expertise with cutting-edge big data technologies in a cloud-native environment.

You will be responsible for creating robust, scalable, and fault-tolerant data pipelines that process massive datasets for both real-time and batch-based analytics. Your work will directly enable our data scientists, analysts, and business teams to derive critical insights and make data-driven decisions.

To adhere to our corporate location policies, this resource will be required to be local to the surrounding Atlanta, GA . You are required to adhere to our Return To Office (RTO) / weekly onsite requirements (Tuesday, Wednesday, and Thursday). 

This position does not offer immigration sponsorship (current or future) including F-1 STEM OPT extension support.

What you’ll do

• Apply the knowledge of data characteristics and data supply pattern, develop rules and tracking process to support data quality model. 

• Prepare data for analytical use by building data pipelines to gather data from multiple sources and systems.

• Integrate, consolidate, cleanse and structure data for use by our clients in our solutions.

• Perform design, creation, and interpretation of large and highly complex datasets.

• Stay up-to-date with the latest trends and advancements in GCP and related technologies, actively proposing and evaluating new solutions.

• Understand best practices for data management, maintenance, reporting and security and use that knowledge to implement improvements in our solutions.

• Implement security best practices in pipelines and infrastructure.

• Develop and implement data quality checks and troubleshoot data anomalies.

• Provide guidance and mentorship to junior data engineers.

• Review dataset implementations performed by junior data engineers.

What experience you need  

• BS degree in a STEM major or equivalent discipline; Master’s Degree strongly preferred

• 5+ years of experience as a data engineer or related role

• Cloud certification strongly preferred

• Advanced skills using programming languages such as Python or SQL and intermediate level experience with scripting languages

• Intermediate level understanding and experience with Google Cloud Platforms and overall cloud computing concepts, as well as basic knowledge of other cloud environments

• Experience building and maintaining moderately-complex data pipelines, troubleshooting issues, transforming and entering data into a data pipeline in order for the content to be digested and usable for future projects 

• Experience designing and implementing moderately complex data models and experience enabling optimization to improve performance

• Demonstrates advanced Git usage and CI/CD integration skills

What could set you apart

• 5+ years of hands-on experience in data engineering or a related field.

• Expert-level proficiency in SQL, Python or relevant programming languages

• Demonstrated experience designing and implementing distributed data processing solutions at scale.

• In-depth, hands-on experience with BigData tools such as  Apache Spark or Apache Beam.

• Proven experience with Google Cloud Platform (GCP) services for data engineering, including Dataflow, Dataproc, BigQuery, and Pub/Sub.

• Experience with workflow orchestration tools like Apache Airflow (or GCP's Cloud Composer).

• Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.

• Background of Large Data Analysis and building AI ML Models would be a huge plus.

• Excellent problem-solving skills and the ability to troubleshoot complex distributed systems.

• Effective communication and collaboration skills, with the ability to articulate technical concepts to both technical and non-technical stakeholders

#LI-Hybrid

#LI-KC1