Member of Technical Staff - Distributed Systems
Sail Research, San Francisco
About the job
Sail builds the world's most efficient software for inference (processing LLM tokens) and agent hosting (cloud VMs). Together, our technologies allow our customers to deploy AI agents at extreme scale for their most ambitious work.
Build the systems that make AI inference fast, reliable, and cost-efficient at global scale. You’ll design the control plane that schedules and autoscales our enormous queue of tokens over a diverse fleet of machines, spread all over the world.
What you’ll do
- Design and implement high-performance schedulers (admission control, queuing, priority, fairness, preemption, bin packing).
- Build global routing and traffic management (latency-aware dispatch, predictive autoscaling, failover strategies, cache-aware routing).
- LLM-specific routing optimizations, e.g. KV caching that lets us trade memory for compute, across the cascading layers of GPU RAM, CPU RAM, and NVMe flash.
- Build deep observability: we want to trace every millisecond of our systems, and catch failures early enough that we can make things right before customers even notice.