Rubrics Expert

• Job Description
• Rubrics Expert
• Contract
• Remote

Job Summary

In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.

Key Responsibilities

• Design, develop, and refine comprehensive rubrics tailored to AI training and assessment needs.
• Analyze task performance and provide structured, actionable feedback to guide improvement and calibration.
• Establish and document effective scoring methodologies that ensure consistency, fairness, and high-quality outcomes.
• Lead calibration sessions to align evaluators and maintain rubric integrity across projects.
• Validate assessment outputs for accuracy, reliability, and alignment with project objectives.
• Oversee quality control processes, making data-driven recommendations for rubric enhancements.
• Collaborate with cross-functional teams to integrate prompt design principles and grocery shopping scenarios into assessment frameworks.

Required Skills and Qualifications

• Proven expertise in rubric design, performance analysis, and feedback structuring for complex assessment environments.
• Strong background in scoring methodology development and calibration for AI or educational projects.
• Demonstrated success in assessment validation and quality control processes.
• Exceptional written and verbal communication skills, with the ability to convey nuanced judgment and rationale.
• Experience with prompt design and rubric creation for AI-driven initiatives.
• Practical, hands-on grocery shopping experience (via apps, delivery, in-store, or pickup) to inform realistic scenario assessments.
• Detail-oriented, analytical, and adaptable approach to evolving project requirements.

Preferred Qualifications

• Previous experience in AI data training or human data evaluation projects.
• Background in instructional design, educational assessment, or related field.
• Familiarity with agile work environments and cross-functional project collaboration.