Company Overview
The company develops an AI-powered revenue operations platform that unifies customer relationship management, sales engagement, conversation intelligence, data enrichment, and pipeline management. Its technology consolidates disconnected business tools into an integrated system designed to improve operational efficiency through automation, analytics, and artificial intelligence.
Job Description
The Data Engineer will build and maintain production-grade Apache Spark pipelines on Databricks that support the company’s analytics products. This hands-on individual contributor role involves working directly with product teams to translate analytical requirements into reliable, cost-efficient data pipelines.
The successful candidate will help ensure data is accurate, available, and delivered within expected freshness requirements. They will also improve pipeline performance, establish monitoring and observability practices, and recommend engineering standards as the data platform scales.
Responsibilities
- Build, deploy, operate, and maintain production Apache Spark data pipelines on Databricks.
- Collaborate with product teams to translate analytics requirements into pipeline designs, including data quality standards and freshness expectations.
- Implement monitoring, observability, and alerting to measure and maintain pipeline reliability.
- Investigate pipeline failures, performance bottlenecks, and excessive infrastructure costs, addressing their underlying causes.
- Optimize Spark workloads and data processing strategies to balance performance, freshness, and cost.
- Evaluate and recommend practical engineering standards for operating data pipelines as the platform and data volumes grow.
- Work independently to resolve ambiguous technical challenges and deliver reliable data solutions.
- Communicate technical trade-offs and pipeline performance considerations to product and engineering stakeholders.
Qualifications
- Approximately 2–6 years of professional experience, consistent with a mid-level to senior individual contributor role.
- Hands-on experience building, deploying, and maintaining production Apache Spark pipelines.
- Strong understanding of Spark performance considerations, including clustering, data shuffles, join strategies, and batch versus stream processing.
- Understanding of how data processing decisions affect performance, data freshness, and infrastructure costs.
- Experience with pipeline reliability, monitoring, and observability.
- Ability to explain practical examples of diagnosing and resolving pipeline failures or performance bottlenecks.
- Strong SQL skills.
- Basic programming proficiency in at least one language; Python is an advantage.
- Ability to work independently, collaborate directly with product teams, and navigate ambiguous requirements.
Preferred Qualifications
- Familiarity with the Databricks ecosystem.
- Experience developing streaming data pipelines and implementing logical replication, such as replicating PostgreSQL data into Databricks.
- Experience building retrieval-augmented generation (RAG) data pipelines.
- Familiarity with AWS infrastructure.
- Experience with platform administration, including roles, permissions, access controls, and visibility management.
Disclaimer
SpringCube curates tech job listings from various company websites to support tech professionals globally.
- No Endorsement: Job ads on SpringCube do not imply endorsement of their authenticity or quality.
- No Client Relationship: This company is not a client of SpringCube unless stated.
- To Apply: Click the “Apply” button to be redirected to the hiring company’s application page for this job.
- No Liability: SpringCube is not liable for inaccuracies.