AI product eval/ AI quality agent/ applied LLM eval/Human-in-the-Loop Evaluation / Annotation Advisor(1-3hr pre week/potential co founder)

Looking for a Part-Time Evaluation Advisor (human annotation)

The direction method is already validated by multiple ai leaders experts and ux researcher, there's early buyer interest /pre pilot from big tech ai team , and the immediate goal is to turn the current build into a small, stable, pilot-ready product.

We're building an early evaluation and workflow product for AI teams. The current focus is helping teams structure and operationalize real-world failure cases, especially in scenarios where an assistant recommends too early, becomes overconfident, or fails to verify what matters before responding. The short-term wedge is a lightweight regression and review workflow. The longer-term opportunity is much bigger: infrastructure for how AI systems are tested, reviewed, and controlled in production decision flows.

Looking for someone who can help with:

• human annotation design
• label guideline writing
• evaluation schema
• translating

Best fit: someone with experience in human eval, annotation design, ranking/review quality, AI eval, or related areas. Bonus if you've worked on shopping, marketplace, trust, search, ranking, or agent systems.

• Part-time / advisor to start. Best fit: someone with experience in human eval, annotation design, ranking/review quality, AI eval, or related areas. Bonus if you've worked on shopping, marketplace, trust, search, ranking, or agent systems.
• Part-time / advisor to start.
• offer equity or cash

Apply to this job