SWE, AI Infrastructure
Genki, Redwood City
About the job
About the role
You will build the platform our ML team works on. Training runs, datasets, experiments, compute, and the path from a checkpoint to a model running on a robot all sit on infrastructure someone has to own.
The measure of this role is iteration speed. When an ML engineer has an idea, how long until they know whether it worked.
What you'll do
- Own the data platform: storage, versioning, and lineage for large multimodal datasets, so any model can be traced back to the data that produced it.
- Build dataset assembly and serving, including the interfaces ML engineers use to slice, filter, and load training data.
- Own our ML compute, on-premise and cloud: cluster provisioning, GPU node health, drivers, networking, and storage.
- Own scheduling and quota across the team, so compute is allocated sensibly and utilization stays high.
- Keep the cluster reliable, including monitoring, failure recovery, and being the person who figures out why a multi-day run died.
- Own training orchestration: job scheduling, multi-node runs, checkpointing, and recovery.
- Build experiment tracking and comparison, so results are reproducible and a run from six months ago can be rerun.