Problem Design Engineer
Idlexperts, United States
About the job
What we’re building
We create high-quality coding problems for training AI models — drawn from real production repositories, not synthetic puzzles. Your role is to ensure these problems are fair, clear, and technically sound.
The role
Part-Time Code Quality Reviewer | Remote | $40/hr | 0-25 hrs/week
We’re looking for engineers who can trace the intent behind a code change, understand what changes were made and why, and express that clearly in natural language.
This is judgment work, not mechanical testing. You'll make philosophical decisions about fairness and clarity in technical contexts.
What you'll actually do
- Review code changes across unfamiliar repositories and tech stacks
- Assess problem fairness: Could a skilled engineer solve this with only the given context?
- Evaluate test quality: Do tests properly validate the intended solution?
- Write clear justifications for your assessments (2-3 sentences per decision)
- Flag coherence issues: Inconsistent artifacts, unclear requirements, missing context
Time commitment: 1-4 hours per problem, fully async — work whenever fits your schedule.