Senior Data Engineer

About the role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer based in Brazil.

We are looking for an experienced Senior Data Engineer to take ownership of a next-generation data platform built around real-time streaming and modern lakehouse technologies. In this role, you will design, develop, and operate scalable data pipelines that transform operational data into reliable, business-critical data products. You will work extensively with Apache Spark Structured Streaming, Change Data Capture (CDC), Apache Iceberg, and AWS infrastructure to deliver data transformations in near real time. Beyond pipeline development, you will oversee compute infrastructure, optimize performance and costs, and ensure the reliability of production systems. You will also build observability tools and improve developer workflows, enabling engineering teams to onboard and manage their own data models. This is an opportunity to influence the evolution of a large-scale data ecosystem while solving complex technical challenges in a distributed, international environment.

Accountabilities: - Streaming Data Pipeline Development: Design, build, and maintain robust Apache Spark Structured Streaming applications that process Change Data Capture (CDC) events into reliable, near-real-time data models.

- Stream Processing Reliability: Implement and optimize stateful processing, slowly changing dimensions (SCD), checkpointing, state stores, late-arriving and partial records, and safe recovery mechanisms to ensure accurate and consistent data processing.

- AWS Infrastructure Management: Own and scale AWS EMR clusters, managing instance sizing, resource allocation, managed scaling, multi-cluster deployments, driver and executor memory optimization, and workload consolidation to improve efficiency and reduce infrastructure costs.

- Lakehouse Management: Maintain and optimize Apache Iceberg tables at scale by designing partitioning strategies and implementing compaction, snapshot expiration, and delete-file cleanup, including specialized maintenance for high-churn tables.

- Data Source Integration: Onboard new business domains onto the streaming platform by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, validating backfills in Snowflake, and coordinating with source database owners.

- Production Troubleshooting: Investigate and resolve complex production incidents across the data platform, addressing issues such as connection failures, offset regressions, token expiration, schema inconsistencies, and write conflicts while focusing on root-cause resolution.

- Monitoring and Observability: Develop monitoring capabilities and diagnostic tools to improve incident detection and resolution, including integrating Spark driver and executor logs with Grafana Loki, implementing watchdog mechanisms, and creating AI-assisted operational runbooks.

- CI/CD and Developer Experience: Improve deployment workflows, configuration management, automated pipelines, and operational safeguards to enable engineering teams to independently and safely deploy new data models.