Data Engineer (Databricks)

About the role

About Onit

We're redefining the future of legal operations through the power of AI. Our cutting-edge platform streamlines enterprise legal management, matter management, spend management and contract lifecycle processes, transforming manual workflows into intelligent, automated solutions.

We’re a team of innovators using AI at the core to help legal departments become faster, smarter, and more strategic. As we continue to grow and expand the capabilities of our new AI-centric platform, we’re looking for bold thinkers and builders who are excited to shape the next chapter of legal tech.

If you're energized by meaningful work, love solving complex problems, and want to help modernize how legal teams operate, we’d love to meet you.

Position Summary

The Data Engineer builds and operates the Databricks-based data pipelines that power a new customer-facing benchmarking capability in Onit's AI-driven legal operations platform. Integrating data from multiple source systems across enterprise legal management, matter management, and spend management, this role turns inconsistent, duplicated, and incomplete source data into a clean, reconciled, and trustworthy foundation for analytics. Working closely with the Data Architect, Data Scientist, and Product, the Data Engineer delivers reliable ingestion, entity resolution, and data quality measurement at scale, with multi-tenant data isolation and privacy built in.

Key Responsibilities • Pipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.

• Data Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.

• Entity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.

• Data Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.

• Orchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.

• Governance & Privacy: Apply Unity Catalog governance, multi-tenant data isolation, and anonymisation and aggregation rules for pooled cross-customer analytics.

• Collaboration: Partner with the Data Scientist to productionise comparability models and data quality metrics, and with Engineering to expose curated data to the product.

Required Skills • 5+ years in data engineering or analytics engineering, building production data pipelines in a cloud environment.

• Hands-on Databricks experience: PySpark and Spark SQL, Delta Lake, Databricks Workflows/Jobs, and Unity Catalog.