Senior / Staff AI Engineer — Evals & Model Quality
FurtherAI, San Francisco
What you will do
- Own evaluations across FurtherAI: Build the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of our AI agents across workflows, Assistant, document processing, and computer use.
- Build high-quality datasets: Develop golden datasets, benchmarks, and regression suites, working closely with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases.
- Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses.
- Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.
- Make development evaluation-driven: Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.