Member of Technical Staff, ML Inference Engineering

This job was selected by the SpringCube team to help AI, Data and Cloud Engineers discover relevant San Francisco Bay Area employers. Sign up to view the full employer details and apply directly with the hiring company.

Company Overview

A rapidly growing Silicon Valley-based AI company focused on transforming human communication through real-time speech and language technology. Its platform applies artificial intelligence to accent translation, noise cancellation, speech enhancement, cross-language communication, and other speech-related applications.

The company combines advanced AI model research with systems engineering and product development to deliver its technology entirely in-house. Its engineering teams work on demanding real-time AI workloads and build infrastructure designed for high performance, reliability, and scalability.

Responsibilities

Inference Optimizations

• Implement custom kernels and low-level optimizations to maximize GPU compute performance.
• Apply graph optimization, operator fusion, and hardware-specific code generation at the ML compiler level.
• Profile, analyze, and resolve deep system bottlenecks to significantly improve latency, throughput, and memory efficiency.
• Drive model-level execution improvements using mixed precision and advanced quantization strategies.

Serving Optimizations

• Develop and optimize custom inference server backends capable of handling complex business logic, scripting, and state management.
• Own and scale model-serving infrastructure across multi-GPU and multi-node deployments while meeting strict on-premise latency requirements.
• Build robust and fault-tolerant runtime services with comprehensive performance benchmarking and monitoring.

Qualifications

• 8+ years of experience writing high-quality, high-performance software, including at least 5 years focused on machine learning systems.
• Strong programming experience with at least one of C, C++, or Rust, along with Python.
• Deep familiarity with modern NVIDIA GPU architectures, such as Ada Lovelace and Blackwell, CUDA, and low-level system profiling.
• Hands-on experience with ML compilers and optimization frameworks such as Apache TVM, TorchInductor/Dynamo, TensorRT, or Triton.
• Experience building or extending model-serving infrastructure, including custom C++ backends for Triton Inference Server or similar serving engines.
• Strong understanding of the AI serving stack, from GPU kernels and quantization through scheduling, state management, and autoscaling.
• Demonstrated history of shipping research or systems that have been adopted or built upon by others in industry or research environments.

Nice-to-Have Qualifications

• Experience serving low-precision FP4 or FP8 models.
• Experience supporting multiple LoRA adapters within a single model instance, including Multi-LoRA architectures.
• Experience deploying models distributed across multiple GPU nodes.
• Research-oriented or systems experience in speech technologies such as speech-to-text (STT), text-to-speech (TTS), speech-to-speech (S2S), or LLM inference.
• Experience operating large-scale on-premise AI training or inference clusters.
• Experience with bare-metal provisioning and Kubernetes management.
• Familiarity with high-performance networking technologies such as InfiniBand or RoCE.
• Experience with distributed storage systems and hardware health monitoring.
• Experience maintaining or contributing to open-source machine learning or systems projects.

Disclaimer

SpringCube curates tech job listings from various company websites to support tech professionals globally.

1. No Endorsement: SpringCube does not endorse or recommend any company or position listed.
2. No Client Relationship: Listing a position does not mean SpringCube has a client, partnership, or employment relationship with the hiring company.
3. To Apply: Use the Apply button to apply directly through the relevant hiring company’s application process.
4. No Liability: SpringCube is not responsible for the hiring process, employment terms, recruitment decisions, or outcomes associated with any listed position.