AI Response Evaluator

About the Role

iMerit is seeking highly detail-oriented Analysts to evaluate and rank AI-generated responses to image-based prompts. In this crucial role, you will meticulously judge AI outputs based on a comprehensive set of criteria, including accuracy, relevance, clarity, conciseness, safety, localization, and adherence to user instructions. A core aspect of the position involves verifying the congruence between an image and the AI model's description, providing written explanations for your judgments.

Key Responsibilities

• Interpret conversational context to accurately discern user intent.
• Rate and rank AI responses against predefined quality criteria.
• Compare multiple AI answers, providing clear written justifications for preferred choices.
• Verify factual claims made by AI models using approved research sources.
• Proactively flag tasks that cannot be reliably assessed to ensure data integrity.

What You'll Bring

• Strong critical thinking abilities and sound judgment, particularly in ambiguous scenarios.
• Solid research skills coupled with exceptional attention to detail.
• Excellent reading comprehension skills.
• Demonstrated self-direction and the discipline to consistently meet deadlines without direct supervision.

Project Details

• Engagement Type: Independent contractor for the duration of the project.
• Location: Fully remote.
• Flexibility: Choose your own hours, provided volume targets and deadlines are met.
• Workload: Task volume is subject to project demand fluctuations.