Responsibilities

Evaluate AI-generated coding interactions end to end.
Judge whether AI outputs are useful, correct at a high level, and aligned with strong engineering practice.
Assess the quality of explanations, reasoning, preambles, and overall user guidance.
Distinguish between different levels of response quality and help define what excellent interactions look like.
Write detailed, direct, and opinionated feedback about what worked, what did not, and what felt misleading.
Record short video explanations of evaluation feedback.

Requirements

Senior, staff, or principal-level software engineering experience, or equivalent experience.
Strong background in TypeScript, JavaScript, or Python.
Hands-on experience with at least one of OpenAI Codex, Claude Code, or Cursor.
Deep familiarity with modern AI-assisted development workflows.
Ability to evaluate code without executing it or reviewing every line.
Strong written and spoken English at B2 level or above.
Comfort giving direct, opinionated feedback and making subjective but rigorous judgments.
High standards for engineering quality.
Prior exposure to prompt design or evaluation workflows is preferred.
Experience mentoring senior engineers or defining engineering standards is preferred.

Remote worldwide contract engagement.
$100-$200 per hour depending on experience and location.
Flexible schedule of 10-20 hours per week.
Ongoing engagement with projects lasting approximately two weeks to a few months.
Successful evaluators may receive new projects.
Start as soon as the take-home exercise is cleared and a project seat is available.
Selection process consists of one take-home evaluation exercise with a recorded Loom walkthrough and no interview.