Senior Applied Scientist, Frontier AI Assets - Assessments (FAI)

In this role you will build the automated systems that make quality assessment at scale. Large language model (LLM)-as-a-Judge is the starting point, and our goal is to develop an agentic system that plan its own audits, critiques its own judgments, and improves its own accuracy over time. You will design it, calibrate it against expert human judgment, and set the technical direction for how the organization measures data quality.

FAI Assessment is part of Frontier AI Assets in AGI. We assess the quality of the datasets and benchmarks behind Amazon's frontier models and agents, and we define what good data means. Today that work relies on human-in-the-loop review by domain experts. Our aim is to automate it with self-improving agents that experts keep calibrated. We are a small team of scientists working closely with the data, modeling, and agent teams.