Data Scientist – Survey Design, Data Annotation, and Machine Learning Evaluation

The SpringCube team curated the following job opportunity to help you in your job search. Explore the position below to find your next career move.

Company Overview

A leading global technology company is developing innovative user-facing conversational experiences powered by state-of-the-art multimodal foundation models. The organization focuses on advancing generative AI capabilities through large-scale data generation, human annotation processes, model evaluation frameworks, and machine learning research. By combining cutting-edge AI technologies with rigorous data science methodologies, the company continues to drive innovation in intelligent products and services used worldwide.

The organization is seeking a Data Scientist – Survey Design, Data Annotation, and Machine Learning Evaluation to join its Machine Learning Evaluations team. This role will focus on managing and analyzing human and automated data annotation processes, developing evaluation methodologies, and improving Large Language Model (LLM) judges used in generative AI model assessment.

The successful candidate will work closely with machine learning engineers and cross-functional stakeholders to design annotation workflows, conduct statistical analysis, optimize prompts, and build reliable evaluation systems for advanced AI models. This position is ideal for professionals with expertise in survey design, data annotation, statistical analysis, and generative AI evaluation.

Key Responsibilities

• Collaborate closely with Machine Learning Engineers to understand data annotation requirements and evaluation objectives.
• Design and manage human data annotation processes, including annotation guidelines, workflow development, and process optimization.
• Develop and refine LLM auto-judges and evaluation criteria for generative AI model assessment.
• Analyze annotation datasets to evaluate model performance and improve automated judging systems.
• Create user instructions and documentation to support high-quality annotation outcomes.
• Monitor and enhance annotation pipeline efficiency and data quality.
• Apply statistical analysis techniques to interpret results and support model evaluation efforts.
• Contribute to the development of scalable evaluation frameworks for multimodal AI systems.

Required Qualifications

• Bachelor’s or Master’s degree in Data Science, Statistics, or a quantitative social science field.
• 2+ years of hands-on experience in survey design and human data annotation.
• Proficiency in Python.
• Strong statistical analysis and data interpretation skills.
• Excellent written and verbal communication skills.
• Experience collaborating with cross-functional teams in technical environments.

Preferred Qualifications

• PhD in Data Science, Statistics, or a quantitative social science field.
• Industry experience conducting product-focused statistical analysis.
• Experience working with large-scale multimodal datasets and annotation pipelines.
• Hands-on experience with LLM prompt engineering and prompt optimization.
• Experience developing or evaluating LLM auto-judges for generative AI systems.
• Record of technical publications, research contributions, or conference presentations in Data Science or related fields.
• Strong cross-functional collaboration and stakeholder management skills.

Disclaimer

SpringCube curates tech job listings from various company websites to support tech professionals globally.

1. No Endorsement: Job ads on SpringCube do not imply endorsement of their authenticity or quality.
2. No Client Relationship: This company is not a client of SpringCube unless stated.
3. To Apply: Click the Apply button to be redirected to the hiring company’s application page for this job.
4. No Liability: SpringCube is not liable for inaccuracies.