Research Engineer, Benchmarking - Member of Technical Staff

Choosing between algorithmic strategies for multi-step LLM work is a measurement problem, and most teams solve it badly: comparisons run case by case, by whoever needs them that week, on whatever task is closest to hand. That doesn't scale, and it doesn't hold up to outside scrutiny - from a customer, or from a reviewer. Callosum needs one benchmarking system: reproducible, contamination-controlled, and trusted enough to be the evidence that decides which approach ships.

This role owns that system. You will build a harness that measures task success, quality, and robustness across motifs, agent topologies, and decomposition strategies, grounded in execution - real commits, real traces, sandboxed grading - rather than self-reported or model-graded scores. The results become the proof points we show customers, the evidence behind the benchmarks we co-publish, and the basis on which an approach ships or doesn't.

This is a research hire that builds. We expect the rigour of a strong evaluation paper applied to a production system, and the engineering ability to design, build, and curate it yourself rather than hand it off.