Senior distributed systems engineer

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work. Overview of the role: The Senior Distributed Systems Engineer position focuses on working with distributed systems technologies including Apache Ozone, Apache Spark, and Apache Kafka to build enterprise-grade platforms designed for massive scale. This role involves architecting and implementing resilient solutions for distributed data processing, storage, and messaging systems that handle petabytes of data across thousands of nodes while contributing to open-source communities. Key responsibilities: - Designing and implementing core features for distributed systems including Apache Ozone, Apache Spark, and Apache Kafka. - Contributing code and design documentation to Apache open-source communities. - Supporting enterprise customers running large-scale big data analytics and ML/AI pipelines. - Partnering with engineering leadership, product managers, and cross-functional groups to understand requirements and turning them into solid design and implementation. - Architecting resilient and scalable solutions for distributed data processing at massive scale. - Developing advanced monitoring, debugging, and performance analysis tools for large-scale distributed systems. - Providing technical leadership for development of streaming messaging products and related technologies. - Leading and mentoring talented groups of engineers working on distributed systems features. Requirements: - Education: Bachelor's degree in Computer Science or equivalent with 5+ years of experience; OR Master's degree with 4+ years of experience. - Programming Expertise: Strong backend engineering skills with expertise in Java, and/or strong proficiency in Scala across modern ecosystems. - Systems Design: Proven experience with large-scale distributed systems design and development. - Technical Foundation: Strong understanding of scaling, replication, consistency, high availability, and system software design. - Architecture Knowledge: Deep understanding of computer architecture, storage, network, and IO subsystems. - Development Skills: Hands-on programming abilities with strong data structures and algorithms expertise. - Code Quality: Passionate about programming with clean coding habits, attention to detail, and focus on quality. - Communication: Strong oral and written communication abilities. - Language Proficiency: Advanced level of English. Additional skills preferred: - Distributed Storage: Background in distributed storage systems (file systems, database storage internals, No SQL storage, or distributed hash tables). - Performance Optimization: Experience in performance tuning and identifying performance bottlenecks. - Open Source: Recognized contributions to open-source projects. - Apache Ecosystem: Experience with Apache Big Data ecosystem projects (Hive, Pig, Map Reduce, HBase, Spark, Kafka). - Specialized Knowledge: Understanding of storage development, RAFT replication frameworks, distributed query processing, and consensus algorithms. - Cloud Platforms: Experience with cloud services (AWS, Azure, GCP, Open Shift). - Dev Ops: Experience with delivery pipeline tools, CI/CD solutions, and automated testing. - Containerization: Experience with Kubernetes and Docker technologies. - Additional Languages: Proficiency in Python, Java Script, or C++. What to expect from us: - Competitive Compensation: Excellent