Job Description
The SpringCube team curated the following job opportunity to help you in your job search. Explore the position below to find your next career move.
Company Overview
An applied AI research laboratory is building end-to-end software agents designed to reason and perform real-world tasks. The organization is known for developing Devin, an AI software engineer, and brings together a highly experienced team with backgrounds in competitive programming, entrepreneurship, and leading AI and technology organizations.
What You’ll Accomplish
Distributed Training Infrastructure
- Build and own systems that reliably run large-scale training jobs across GPU clusters.
- Develop job launchers, checkpointing and recovery systems, fault-tolerance mechanisms, and monitoring tools.
- Ensure researchers remain informed and unblocked throughout training runs.
Scaling Agent Rollouts
- Own infrastructure supporting hundreds of thousands of concurrent coding-agent rollouts within VM sandboxes.
- Develop high-fidelity environments and distributed systems capable of supporting large-scale reinforcement learning workloads.
Performance Optimization
- Profile and improve training throughput across the full system.
- Identify bottlenecks involving data loading, communication overhead, memory utilization, and compute efficiency.
- Implement solutions that improve step time and Model FLOPs Utilization (MFU) at scale.
Experiment Orchestration and Tooling
- Design and maintain systems used to launch, track, and analyze research experiments.
- Reduce friction within the research cycle and enable researchers to spend more time developing and testing ideas.
Data Pipeline Engineering
- Build high-throughput and reliable data pipelines for training and evaluation.
- Maintain data quality, reproducibility, and efficiency at the scale required by large training runs.
Debugging and Reliability
- Diagnose and resolve failures involving GPUs, networking, numerical issues, and data.
- Maintain a detailed understanding of system failure modes and develop infrastructure that can recover quickly and fail gracefully.
Parallelism and Systems Research
- Implement and optimize data, tensor, pipeline, and sequence parallelism strategies.
- Evaluate the tradeoffs of different approaches and apply them to maximize available hardware performance.
Scaling Infrastructure Ahead of Research
- Anticipate future infrastructure requirements and build solutions before they become constraints.
- Proactively identify potential bottlenecks and develop systems that enable research teams to scale efficiently.
Exceptional Candidates Have
- Deep experience building and operating distributed training systems for large models, with the ability to own infrastructure from the cluster level through the training loop.
- Strong systems engineering fundamentals across distributed systems, networking, storage, and full-stack hardware and software performance.
- Proficiency in Python and C++.
- Experience with PyTorch or equivalent deep learning frameworks at the systems level rather than solely through API usage.
- Hands-on experience with GPU performance profiling, memory optimization, and compute efficiency.
- Ability to diagnose and resolve performance issues affecting large-scale training workloads.
- Experience implementing or optimizing data, tensor, pipeline, and sequence parallelism strategies.
- A track record of developing tooling and abstractions that significantly accelerate research workflows.
- Strong debugging abilities across complex distributed systems where failures may be non-deterministic and difficult to reproduce.
- Sufficient machine learning knowledge to collaborate effectively with researchers and understand training objectives, architectural decisions, and infrastructure requirements.
- Demonstrated technical capability and problem-solving ability. Advanced academic credentials may be beneficial but are not the sole measure of suitability.
Resources & Environment
- Work within a small, highly selective team where research and product development move closely together.
- Contribute to prototypes that can progress rapidly toward real-world deployment.
- Own and operate infrastructure spanning thousands of GPUs.
- Work in an environment where compute resources and access to required systems are available to support ambitious research.
- Operate with a high degree of autonomy, technical depth, and speed.
- Work with minimal process overhead in a fast-moving and highly competitive AI environment.
Disclaimer
SpringCube curates tech job listings from various company websites to support tech professionals globally.
- No Endorsement: Job ads on SpringCube do not imply endorsement of their authenticity or quality.
- No Client Relationship: This company is not a client of SpringCube unless stated.
- To Apply: Click the Apply button to be redirected to the hiring company’s application page for this job.
- No Liability: SpringCube is not liable for inaccuracies.
