Lead Data Engineer

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work. Overview of the role: The Lead Data Engineer position focuses on serving as core contributors to internal Data Platforms, responsible for building scalable, resilient pipelines that support Business Operations, Analytics, and cutting-edge AI/ML initiatives. This role involves designing, implementing, and optimizing data pipelines using a wide range of Big Data and cloud-native technologies while providing technical leadership, guiding personnel, and ensuring architectural consistency across projects. Key responsibilities: - Providing technical leadership and mentorship to other data engineers, helping them develop and succeed in their roles. - Leading design reviews, enforcing coding standards, and guiding architectural decisions to ensure platform scalability and maintainability. - Partnering with senior stakeholders and engineering leadership to prioritize and plan technical work that aligns with strategic business goals. - Acting as technical points of contact for cross-team collaborations, participating in project planning, scoping, and retrospectives. - Demonstrating curiosity, initiative, and capability to integrate AI-powered tools into day-to-day engineering workflows. - Supporting AI/GenAI projects by building pipelines that feed LLM apps and recommendation systems. - Driving adoption of data engineering best practices across organizations, including testing, CI/CD, and observability. - Designing, building, and maintaining batch and streaming data pipelines using native data platforms. - Ingesting data from various sources, including REST APIs, cloud services, and enterprise SaaS platforms. - Implementing robust transformation logic using Python, PySpark, SQL, and Java across structured and semi-structured datasets. - Ensuring data integrity, lineage, and performance across ingestion, transformation, and delivery layers. Requirements: - Data Engineering Experience: Substantial background (6+ years) as Data Engineer in enterprise-scale data infrastructures. - Leadership Skills: Proven leadership in technical project management and personnel mentorship. - Architectural Guidance: Track record of guiding architectural conversations in high-throughput data ecosystems. - Programming Mastery: Advanced proficiency in SQL and Python programming, with expertise in PySpark or Java. - Data Architecture: Comprehensive knowledge of data modeling, transformation methodologies, and data warehouse architectures. - Hadoop Ecosystem: Proven expertise with Hadoop ecosystem technologies. - Pipeline Development: Successful implementation of resilient, high-performance data pipelines meeting strict service level agreements. - Integration Expertise: Deep understanding of API integration, performance optimization, and data schema adaptation. - Language Proficiency: Advanced level of English. Additional skills preferred: - Workflow Tools: Practical implementation experience with Apache NiFi and Apache Airflow. - Development Environment: Expertise in containerized environments, developer tooling, and API frameworks. - AI/ML Platforms: Demonstrated capabilities in AI/ML platform support and generative AI pipeline development. - Innovation Contributions: Proven contributions to internal technological innovations within data engineering workflows. What to expect from us: - Competitive Compensation: Excellent