Data Platform Lead

About Us

We are building a state-of-the-art, on-premise data platform from the ground up to revolutionize how our organization leverages data. Inspired by a unified and modern approach, our mission is to create a single, cohesive ecosystem that eliminates silos between data engineering, business analytics, and machine learning. We are looking for a key contributor to our team who is excited by the challenge of turning this vision into a scalable, resilient, and performant reality that will serve as the engine for our data-driven future.

Job Summary

As a Platform Engineer, you will be a core contributor to the team building and scaling our new on-premise data platform. You are a hands-on engineer who is passionate about backend systems, container orchestration, and automated infrastructure. You will be responsible for implementing and managing the foundational layers of the platform, directly enabling our Data Engineers, Data Scientists, and Analysts to work seamlessly. This role is a unique blend of backend software engineering, DevOps principles, and Kubernetes expertise. Depending on your experience, you will either grow into a subject matter expert or lead the charge in designing solutions to the complex challenges of scalability, multi-tenancy, and performance outlined in our strategic vision.

What You'll Do

• Build and Manage the Foundation: Implement, manage, and ensure the reliability of a highly available, hardened Kubernetes cluster, which serves as the "distributed operating system" for all data services.
• Automate Everything: Utilize Infrastructure as Code (IaC) tools like Terraform to automate platform setup, from network configurations to user policies and service deployments.
• Deploy and Operate Data Services: Containerize, deploy, and manage the lifecycle of a curated suite of open-source data services, including Apache Spark, Trino, MinIO, and Apache Airflow.
• Secure the Platform: Implement and manage secure and high-performance cluster networking (using a CNI like Calico or Cilium) and contribute to the platform's security posture through RBAC, network policies, and secrets management.
• Enable Developer and User Success: Build and maintain CI/CD pipelines for platform components, and help create the underlying environments (e.g., Jupyter notebooks, SQL query engines) that our data personas will use every day.
• Tackle Scaling Challenges: Work with the team to solve complex distributed systems problems, including addressing performance bottlenecks in data shuffling, object storage, and metastore lookups.
• Ensure Reliability: Contribute to the implementation of comprehensive monitoring, logging, and alerting for all platform layers to ensure stability, manage costs, and facilitate quick troubleshooting.
• Collaborate: Partner with Data Engineers and Data Scientists to understand their needs, troubleshoot issues, and provide a stable and efficient platform that accelerates their work.

Core Qualifications (Mid-Level)

• Solid hands-on experience deploying and managing applications on Kubernetes.
• Proficiency with container technologies (Docker, containerd).
• Experience writing Infrastructure as Code (IaC) using tools like Terraform or Ansible.
• Strong backend software development skills in a language like Python, Go, or Java.
• Experience with building or maintaining CI/CD pipelines (e.g., Jenkins, GitLab CI).
• A good understanding of distributed systems concepts, including networking and storage fundamentals within a Kubernetes context.
• A security-conscious approach to building and managing infrastructure.

Senior-Level Qualifications (What Sets You Apart)

• Proven experience designing and architecting production-grade Kubernetes clusters, especially in on-premise or hybrid environments.
• Expertise in solving complex, non-obvious scaling issues within distributed systems (e.g., you can speak to the "shuffle problem" or have dealt with "chatty" protocols at scale).
• A track record of leading the implementation of secure, high-performance cluster networking and storage solutions (CNI, CSI).
• Experience deploying and managing components of the modern data stack (e.g., Spark, Trino, Kafka, MinIO, Airflow).
• Ability to mentor other engineers and lead technical decision-making for the team.
• Deep experience in observability and performance tuning for both Kubernetes and the applications running on it.