Databricks Data Engineer | Senior

About the role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Databricks Data Engineer based in Brazil.

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable, high-performance data engineering solutions. In this role, you will leverage the Databricks ecosystem to build modern Lakehouse architectures and implement Medallion data models (Bronze, Silver, and Gold). You will be responsible for developing robust data pipelines, integrating multiple data sources, and enabling efficient data processing through technologies such as PySpark, Spark SQL, and Delta Lake. Working with Azure cloud services, you will ensure data quality, governance, security, and operational reliability across production environments. As a technical reference for the team, you will guide engineering best practices, support architectural decisions, and mentor other professionals. This is an opportunity to contribute to advanced data platforms while driving innovation, performance, and cost efficiency in a collaborative technology environment.

Accountabilities: - Design, develop, implement, and continuously improve data engineering solutions using the Databricks ecosystem.

- Define and implement scalable data architectures based on Lakehouse principles and the Medallion architecture (Bronze, Silver, and Gold layers).

- Develop, optimize, and maintain data pipelines using Databricks, PySpark, Spark SQL, Apache Spark, and Delta Lake.

- Implement data ingestion, processing, and integration solutions, including Change Data Capture (CDC) and Structured Streaming.

- Build and manage data pipeline orchestration and system integrations using Azure Data Factory (ADF).

- Develop REST APIs, microservices, and automation solutions using Python.

- Establish and maintain data governance, security, lineage, and access control standards through Unity Catalog.

- Optimize data processing performance, scalability, and infrastructure costs through efficient resource allocation and workload management.

- Apply advanced performance tuning techniques, including Photon optimization, partitioning, shuffle management, and execution plan analysis.

- Implement and maintain observability, monitoring, and operational support practices for production data pipelines.

- Manage Databricks Workflows, CI/CD processes, Git repositories, and Declarative Automation Bundles.

- Identify and resolve complex technical challenges while making architectural decisions aligned with business and engineering requirements.

- Act as a technical reference for the data engineering team, mentoring professionals, sharing knowledge, and establishing engineering best practices.

Requirements:

- Extensive professional experience delivering and maintaining production-grade data engineering solutions using Databricks.