Lead Data Engineer

About the role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer based in Brazil.

We are looking for a hands-on Lead Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. In this role, you will work extensively with Python, PySpark, and AWS technologies to deliver reliable, production-ready data processing systems. You will oversee workflow orchestration, infrastructure automation, and CI/CD practices while ensuring strong data quality and operational performance. Working closely with engineering, product, and business teams, you will contribute technical leadership and promote effective development practices. This opportunity combines hands-on engineering with team leadership in an Agile, collaborative environment.

Accountabilities: -

Design, develop, and maintain scalable data pipelines running on AWS infrastructure.

-

Build reliable data processing solutions using Python, Pandas, and PySpark.

-

Configure and manage data workflows using AWS Managed Workflows for Apache Airflow (MWAA) or Apache Airflow.

-

Develop and optimize queries with AWS Athena to support efficient data analysis.

-

Automate infrastructure provisioning and deployment processes using Terraform and GitHub Actions.

-

Implement testing, monitoring, troubleshooting, and data-quality controls to ensure pipeline reliability.

-

Participate in code reviews and promote engineering best practices across the team.

-

Collaborate with engineering, product, and business stakeholders in an Agile environment.

-

Provide technical guidance and leadership while supporting the development team's delivery and performance.

Requirements:

-

At least 5 years of relevant professional experience.

-

At least 1 year of experience leading and managing development teams.

-

Strong Python programming skills applied to data engineering.

-

Practical experience using Pandas for data manipulation and analysis.

-

Advanced working knowledge of PySpark for large-scale distributed data processing.

-

Experience with AWS MWAA or Apache Airflow for workflow orchestration.

-

Familiarity with AWS Athena for querying and analyzing data stored in AWS.

-

Hands-on experience with Terraform for infrastructure automation and management.

-

Working knowledge of GitHub Actions and automated CI/CD pipelines.

-

Good understanding of DevOps principles and their application to data engineering.

-