Member of Technical Staff — Data Infrastructure
Causal Labs, San Francisco +1
What you will do
- Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
- Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on
- Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
- Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
- Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
- Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match
- Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly
