Data Engineer (Databricks + Fabric)

About the role

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Data Engineer (Databricks + Fabric) based in Brazil.

In this role, you will design, build, and operate production-ready data platforms for clients, transforming complex data requirements into scalable and reliable solutions. You will work hands-on with Databricks and Microsoft Fabric to develop advanced ETL/ELT pipelines, implement modern lakehouse architectures, and optimize data processing workflows. Using technologies such as PySpark, Delta Lake, Python, and SQL, you will help organizations improve data accessibility, quality, governance, and operational efficiency. You will also implement Medallion architecture principles and establish robust data governance practices through Unity Catalog. Working closely with clients and cross-functional teams, you will identify technical challenges, recommend improvements, and contribute to new data engineering initiatives. This is an opportunity to take ownership of impactful production environments while working remotely in an international technology consulting environment.

Accountabilities: - Design, develop, deploy, and maintain production-grade data platforms for clients, taking hands-on responsibility for implementation and ongoing operations.

- Build and optimize scalable ETL/ELT pipelines using Databricks and Microsoft Fabric to support complex data integration and transformation requirements.

- Design and implement modern lakehouse architectures based on the Medallion framework, organizing data into Bronze, Silver, and Gold layers using Delta Lake.

- Develop and maintain data processing solutions using PySpark, Python, SQL, Databricks Workflows, and Microsoft Fabric components.

- Implement and manage data governance through Unity Catalog, including permissions, access controls, data lineage, and data quality practices.

- Work across the Microsoft Fabric ecosystem, leveraging Lakehouse, OneLake, Data Pipelines, Notebooks, Dataflows Gen2, and Warehouse capabilities.

- Establish and automate deployment processes using CI/CD pipelines and Infrastructure as Code (IaC) practices.

- Ensure data platforms are scalable, secure, reliable, and optimized for performance and maintainability.

- Gather, analyze, and translate business and technical requirements in collaboration with clients and internal teams.

- Identify opportunities to improve existing data architectures, infrastructure, and engineering workflows.

- Proactively recognize potential client needs and recommend technical enhancements that may lead to additional projects and engagements.

- Collaborate with cross-functional teams to ensure successful project delivery and alignment with business objectives.

Requirements:

- Proven hands-on professional experience implementing and managing Databricks solutions in production environments. Proofs of concept or theoretical knowledge alone are not sufficient.