Applied AI Researcher, Agent Systems & Evaluation

This job was selected by the SpringCube team to help AI, Data and Cloud Engineers discover relevant San Francisco Bay Area employers. Sign up to view the full employer details and apply directly with the hiring company.

Company Overview

The company is a self-driving technology organization focused on making autonomous transportation accessible at scale. Founded in 2016, it develops autonomous driving technology that combines advanced artificial intelligence with automotive-grade hardware. Its technology supports applications including robotaxis, commercial fleets, and personally owned vehicles, providing automakers and mobility platforms with a pathway toward large-scale autonomous vehicle deployment.

About the Team

The team focuses on building the infrastructure needed to make AI agent systems reliable, measurable, and trustworthy. Rather than relying primarily on intuition or anecdotal improvements, the team develops rigorous evaluation, verification, and measurement systems that determine whether autonomous AI outputs can be trusted.

Key Responsibilities

Own the Evaluation Pipeline End to End

The role encompasses three interconnected stages:

Eval Data Collection

• Identify and establish reliable sources of ground-truth evaluation data.
• Mine production traces to identify and label outcomes.
• Capture human accept, reject, and edit signals as they occur.
• Build task sets that accurately represent the real distribution of engineering work.
• Evaluate when model-based judges can provide reliable signals and when they may introduce unsupported assumptions.

Eval Loop Construction

• Convert ambiguous objectives into measurable evaluations that can run consistently across system changes.
• Establish appropriate noise floors and statistical standards for evaluating improvements.
• Develop task suites designed to resist metric gaming.
• Design experiments suitable for production environments where controlled randomization may not always be possible.
• Build evaluation processes that provide reliable evidence for agent-system development decisions.

Automated Hill Climbing

• Use reliable evaluation loops to systematically search for improvements rather than relying solely on manual design.
• Optimize areas such as prompts, context strategies, tool configurations, routing, reasoning budgets, and eventually model selection.
• Ensure automated optimization is grounded in trustworthy evaluation signals so that improvements are tied to meaningful outcomes.

Post-Train Models on Proprietary Data

• Conduct hands-on model research involving supervised fine-tuning and reinforcement learning.
• Work with open-source vision-language models.
• Develop training approaches using proprietary autonomous-driving data collected through years of real-world operations.
• Collaborate with an available labeling workforce to define and generate the specific datasets required for research hypotheses.
• Develop hypotheses about model behavior, commission targeted data collection, post-train models, and evaluate their performance against real-world driving outcomes.

Qualifications

• Experience researching or developing AI systems, agent systems, evaluation frameworks, or machine learning models.
• Strong understanding of transformer-based models and modern AI development practices.
• Experience with evaluation methodology, experimentation, and quantitative measurement.
• Experience working with supervised fine-tuning and/or reinforcement learning is valuable.
• Experience with open-source vision-language models is valuable.
• Ability to design datasets and evaluation tasks that accurately represent real-world workflows.
• Strong ability to translate ambiguous objectives into measurable experiments and evaluation criteria.
• Experience analyzing production data, user feedback, or human-generated signals is beneficial.
• Ability to work independently in an environment where research direction and technical decisions require significant ownership.

Disclaimer

SpringCube curates tech job listings from various company websites to support tech professionals globally.

1.No Endorsement: Job ads on SpringCube do not imply endorsement of their authenticity or quality.
2.No Client Relationship: This company is not a client of SpringCube unless stated.
3.To Apply: Click the “Apply” button to be redirected to the hiring company’s application page for this job.
4.No Liability: SpringCube is not liable for inaccuracies.