Data Engineer II

About the role

About DAT

DAT is an award-winning employer of choice and a next-generation SaaS technology company that has been at the leading edge of innovation in transportation supply chain logistics for 45 years. ders optimistically share future possibilities to inspire and motivate others toward their full potential. We expect our employees to find ways to embrace positive change, be curious and challenge the status quo, and provide solutions to unmet problems. Joining DAT means joining a culture focused on fostering development, building genuine connections, recognizing each other’s strengths and sharing in successes.

We continue to transform the industry year over year, by deploying a suite of software solutions to millions of customers every day - customers who depend on DAT for the most relevant data and most accurate insights to help them make smarter business decisions and run their companies more profitably. We operate the largest marketplace of its kind in North America, with 400 million freights posted in 2022, and a database of $150 billion of annual global shipment market transaction data. Our headquarters are in Denver, CO, with additional offices in Missouri, Oregon, and Bangalore, India. For additional information, see www.DAT.com/company.

Key Technologies

- Data Warehouse: Snowflake/Data Lake

- Transformation: dbt (data build tool)

- Orchestration: Apache Airflow

- CI/CD: Git, GitHub Actions

- Programming: Python

- AI-Assisted Development: LLM-based coding assistants (e.g., Claude, Copilot) for code generation, review, and documentation

Core Responsibilities

Data Pipeline Development & Engineering

- ELT Process Implementation: Assist senior engineers in designing, building, testing, and maintaining cloud-native ELT (Extract, Load, Transform) data pipelines, ensuring data is reliably loaded into Snowflake and S3 Iceberg layer.

- Pipeline Architecture: Contribute to the design of scalable, modular data pipelines that support incremental loads, backfills, and reprocessing with minimal manual intervention.

- Transformation with dbt: Develop and maintain data models using dbt (data build tool) for data cleaning, aggregation, and transformation and utilize the Snowflake data warehouse.

- Python Scripting: Utilize Python to build custom data extraction scripts, implement monitoring tools, and contribute to general automation efforts.

- Pipeline Reliability: Build in retry logic, alerting, and failure-handling patterns so pipelines degrade gracefully and self-recover where possible.

Workflow Orchestration and Automation

- Airflow DAGs: Learn to author, schedule, and monitor data workflows defined as Directed Acyclic Graphs (DAGs) in Apache Airflow.

- Pipeline Scheduling: Integrate and orchestrate dbt runs and other pipeline tasks within Airflow to manage dependencies and execution timing.