Member of Technical Staff, Evals and Post-Training
Profound, New York, New York +1
What you will do
- Build Profound’s eval system: tasks, rubrics, LLM judges, programmatic verifiers, regression alerts, and self-improvement loops where we look at the traces when something is wrong and come up with the fix
- Define what a great AI Marketer run looks like, and analyze trajectories across multi-step tool use, analytical correctness, and content quality
- Work with marketing experts and data labeling companies to define tasks, metrics, rubrics, and other grading mechanisms
- Close the loop from usage: input use cases, cluster failure modes, and turn these into reliable evals
- Turn eval results and expert feedback into training signal: rewards, preference data, and training environments
- Build tooling for debugging model and agent behavior
- Make evals part of how we decide things: wired into CI, launch reviews, and model selection