Senior Data Scientist AI Evaluation
Alpaca , Remote - Americas +1
What you will do
- Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
- Build repeatable eval loops: Track quality over time and catch regressions before release.
- Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
- Drive iteration: Translate eval results into actionable recommendations for system improvements.
- Establish quality standards: Set evaluation guidelines, documentation, and review practices.
- Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.
What you need
- Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).
- Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.