As the Data Platform Lead, you will lead the design and development of a scalable and versatile data platform with a microservices architecture. This platform will be at the heart of our data science and machine learning initiatives, enabling data scientists to experiment, deploy, and iterate on ML models seamlessly.
Responsibilities
- Platform Architecture: Lead the design and implementation of a robust data platform with microservices architecture, ensuring scalability, reliability, and flexibility.
- Data Ingestion: Oversee the development of data ingestion pipelines to acquire, clean, and preprocess data from various sources, making it readily available for analysis.
- Microservices Development: Build and maintain microservices that enable data scientists to access and manipulate data, develop ML models, and deploy them into production.
- Integration: Collaborate with cross-functional teams to integrate the data platform with other systems, tools, and data sources.
- Performance Optimization: Continuously optimize the platform for performance, cost-efficiency, and resource utilization.
- Security and Compliance: Implement robust security measures and ensure compliance with data privacy regulations.
- Documentation: Maintain comprehensive documentation for platform architecture, APIs, and development processes.
- Team Leadership: Mentor and lead a team of engineers and data professionals, fostering a culture of innovation and excellence.
Requirements
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- Proven experience as a Data Platform Lead or a similar role with expertise in microservices architecture.
- Strong proficiency in cloud platforms (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes).Experience with big data technologies (e.g., Hadoop, Spark, Kafka) and data processing frameworks.
- Proficiency in programming languages (e.g., Python, Java) and data querying languages (e.g., SQL).Excellent problem-solving and architectural design skills.
- Strong communication and leadership skills.
- Experience in machine learning or data science is a plus.