Generalist Annotator-RLHF

Job Description About Us Soul AI by Deccan AI is a pioneering company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in delivering high-quality human-curated data, AI-first scaled operations services, and more. Based in SF and Hyderabad, we are a young, fast-moving team on a mission to build AI for Good, driving innovation and positive societal impact. About the Role You will evaluate and annotate LLM's outputs across multiple modalities like text, images, audio, and video and provide structured human feedback used to train and align State of the Art (SOTA), AI models through Reinforcement Learning from Human Feedback (RLHF). This role requires strong analytical ability, attention to detail, and the judgment to assess correctness, safety, reasoning quality, and compliance with guidelines across diverse tasks. Responsibilities - Compare and rank AI-generated text responses for reasoning quality, clarity, correctness, and instruction-following. - Annotate text tasks: reasoning steps, errors, hallucinations, tone issues, policy violations. - Evaluate and label image tasks (classification, captions, OCR, object detection). - Transcribe and assess audio outputs for accuracy, sentiment, and misinterpretation. - Review video clips for event detection, scene understanding, and summary correctness. - Provide clear feedback explaining why one model output is superior to another (RLHF). - Maintain high consistency with annotation guidelines; flag edge cases and quality issues. - Collaborate with internal teams to refine evaluation criteria. Skills & Experience Required - Strong analytical and critical reasoning skills. - Excellent written communication for explaining judgments. - Experience with text, image, audio, or video annotation tools is a plus. - Ability to detect subtle errors, logic gaps, or policy-violating content. - Comfort with ambiguous tasks and making justified decisions. - Bonus: experience in QA, research, content evaluation, or AI model assessment. Education Qualifications - Bachelor's degree in Humanities, Social Sciences, Engineering, Computer Science, Psychology, Linguistics, or any field requiring analytical writing and critical thinking. - Candidates with strong reasoning and communication skills may be considered regardless of educational background. Why Join Us - Directly influence how multi-modal AI systems learn, reason, and behave. - Work across varied tasks - from text evaluation to image analysis to audio/video interpretation. - Collaborate with researchers and domain experts shaping next-generation AI alignment.