Member of Technical Staff - Inference

In this role, you'll own token processing down to the lowest layers of the stack. You'll do things like: develop a new request scheduling strategy, achieve better communication/computation overlap, investigate novel schemes for increasing cache hit rates, or identify a better way to benchmark inference performance.

• Modify and extend state-of-the-art inference engines like vLLM and SGLang, and work on our own internal engine.
• Understand every microsecond of GPU time spent during a forward pass. You'll be able to explain every kernel launch on an nsys profile.
• Design and implement exotic parallelism schemes to work with "interesting" hardware topologies.
• Write and debug GPU kernels to excel in specific regimes, such as cascade attention https://flashinfer.ai/2024/02/02/cascade-inference.html