Software Engineer, Systems — AI Training Data Infrastructure

About the role

We own the dataset layer that Meta's largest AI training runs read from. AIRStore and the Anywhere Training substrate are what let a multi-petabyte dataset be written once and read at full throughput from any cluster, in any region, in any cloud — without a copy. Our customers are named model programs, not abstract services: when a training job's GPUs go idle waiting on I/O, or a dataset isn't where the scheduler put the job, that is our problem to own and fix. In 2026 this team moved Anywhere Training blob pointers to 100% Manifold residency, cut AIRStore dataset startup time by 10×, drove the migration of AIRStore datasets onto a standard S3 interface, and reclaimed hundreds of petabytes through lifecycle work — all while holding the line on training reliability across dozens of production incidents.

Responsibilities

Own significant components of the AIRStore data path end to end — ingestion, metadata, client, and read path — from design through production operation Attack throughput and latency as a first-class product concern: prefetching, parallelism, caching, and startup cost, measured in GPU utilization and training wall-clock rather than microbenchmarks Build the multi-region and multi-cloud story: make dataset location invisible to the training job, across Meta data centers and third-party clouds Get dataset lifecycle right — TTL, archival, expiration, and deletion — where the correctness bar is absolute in both directions: nothing a live run needs may disappear, and nothing that must be deleted may persist Own a widely embedded client library responsibly: compatibility, rollout safety, and blast-radius control across thousands of callers you don't control Take real operational ownership. Join the oncall rotation, drive root-cause analysis on incidents affecting production model training, and convert each one into a structural fix rather than a mitigation Partner directly with AI research and training teams, Manifold, Warm Storage, Privacy, and Crypto to land changes that cross system boundaries

Minimum Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience 5+ years of experience building and operating production distributed systems or large-scale data infrastructure Proficiency in a systems language — C++, Rust, or Go — plus Python Demonstrated ability to diagnose performance problems in production: profiling, tracing, and reasoning about I/O, network, and concurrency behavior at scale Experience owning a service in production, including oncall, incident response, and postmortem follow-through Track record of designing and delivering a substantial system component with limited direction BS/MS in Computer Science or equivalent practical experience

Preferred Qualifications

Experience with data lifecycle, retention, and privacy-driven deletion at scale Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies Evidence of influencing technical direction beyond your immediate team Background in ML data pipelines — dataloading, checkpointing, dataset versioning, or throughput-bound training I/O Experience with storage systems: object/blob stores, distributed filesystems, caching and prefetch layers, or dataset/columnar formats Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews) Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements) Experience running infrastructure across multiple cloud providers or hybrid environments Familiarity with S3-compatible object storage APIs and the practical tradeoffs of compatibility layers