Job Description
This job was selected by the SpringCube team to help AI, Data and Cloud Engineers discover relevant San Francisco Bay Area employers. Sign up to view the full employer details and apply directly with the hiring company.
Company Overview
A fintech company is transforming private market investing by enabling registered investment advisors (RIAs) and family offices to discover, model, and manage private market investments through a digital platform. The platform combines curated fund and co-investment opportunities with institutional-grade infrastructure, providing access to venture capital, private equity, private credit, and other private asset classes.
Key Responsibilities
Build and Extend Data Architecture
- Build and extend a data lakehouse on AWS combining data lake storage and warehouse technologies for diverse financial datasets.
- Contribute to a knowledge graph representing relationships among investors, funds, companies, and other financial entities.
- Support vector database integrations for semantic search and retrieval across AI agents, models, and providers.
- Design scalable data architectures that support analytics, AI, and product development.
Develop Data Pipelines
- Create robust ETL/ELT pipelines to ingest, clean, transform, and integrate data from internal applications and third-party APIs.
- Develop both batch-processing and real-time streaming capabilities.
- Design pipelines for scalability as data volume and complexity increase.
- Implement appropriate error handling, monitoring, and reliability practices.
- Ensure data pipelines provide accurate and timely information for analytics and recommendations.
Enable AI/ML Capabilities
- Partner with data science and engineering teams to provide the data and infrastructure required for machine learning and AI features.
- Prepare datasets for model training and support the development of feature stores.
- Build workflows that provide LLM-based agents with relevant context through data retrieval and vector similarity search.
- Support systems that deliver AI model outputs and recommendations to products in real time.
- Contribute to AI-enabled data infrastructure and semantic search capabilities.
Engineering Excellence and Collaboration
- Partner with data leadership and engineering teams to deliver data and AI infrastructure.
- Participate in code reviews, testing, architecture discussions, and engineering best practices.
- Help application and service engineers consume data effectively.
- Collaborate with cross-functional squads to incorporate data-driven features into product initiatives.
- Share technical expertise and support the growth of engineering teams.
Infrastructure and DevOps
- Collaborate with DevOps engineers to deploy and maintain data services.
- Containerize and orchestrate data tools using Docker and Kubernetes on AWS EKS.
- Implement CI/CD pipelines for automated testing and deployment of data workflows.
- Monitor the health and performance of data platforms using alerts and dashboards.
- Troubleshoot and resolve production issues affecting critical data and AI services.
- Maintain reliable and highly available data infrastructure.
Continuous Improvement
- Stay current with developments in data engineering, cloud technologies, AI, and machine learning.
- Evaluate emerging AWS services and open-source technologies.
- Assess technologies such as Kafka, Kinesis, and Airflow to improve pipeline reliability and scalability.
Required Qualifications
- 5+ years of hands-on experience in data engineering or related fields.
- Experience designing and building large-scale data pipelines and storage solutions.
- Experience taking data engineering projects through the complete lifecycle from design to production deployment.
- Strong AWS experience with services such as S3, EC2, ECS, EKS, Athena, Redshift, Glue, and Step Functions.
- Experience with infrastructure-as-code tools such as Terraform or CloudFormation is a plus.
- Strong proficiency in SQL and relational database design.
- Experience designing efficient database schemas and optimizing queries and indexes.
- Experience with data warehouses or lakehouses such as Snowflake or Databricks Delta Lake.
- Familiarity with graph databases such as Neo4j or AWS Neptune and knowledge graph schemas.
- Fluency in at least one major programming language used in data engineering.
- Strong Python experience, with pandas and/or PySpark experience preferred.
- Experience with TypeScript or Node.js in data-related applications is valuable.
- Ability to develop clean, maintainable, and well-tested code.
Disclaimer
SpringCube curates tech job listings from various company websites to support tech professionals globally.
1.No Endorsement: Job ads on SpringCube do not imply endorsement of their authenticity or quality.
2.No Client Relationship: This company is not a client of SpringCube unless stated.
3.To Apply: Click the “Apply” button to be redirected to the hiring company’s application page for this job.
4.No Liability: SpringCube is not liable for inaccuracies.**
