Lead Data Engineer

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work.Overview of the role:The Lead Data Engineer position focuses on serving as core contributors to internal Data Platforms, responsible for building scalable, resilient pipelines that support Business Operations, Analytics, and cutting-edge AI/ML initiatives. This role involves designing, implementing, and optimizing data pipelines using a wide range of Big Data and cloud-native technologies while providing technical leadership, guiding personnel, and ensuring architectural consistency across projects.Key responsibilities:- Providing technical leadership and mentorship to other data engineers, helping them develop and succeed in their roles.- Leading design reviews, enforcing coding standards, and guiding architectural decisions to ensure platform scalability and maintainability.- Partnering with senior stakeholders and engineering leadership to prioritize and plan technical work that aligns with strategic business goals.- Acting as technical points of contact for cross-team collaborations, participating in project planning, scoping, and retrospectives.- Demonstrating curiosity, initiative, and capability to integrate AI-powered tools into day-to-day engineering workflows.- Supporting AI/GenAI projects by building pipelines that feed LLM apps and recommendation systems.- Driving adoption of data engineering best practices across organizations, including testing, CI/CD, and observability.- Designing, building, and maintaining batch and streaming data pipelines using native data platforms.- Ingesting data from various sources, including REST APIs, cloud services, and enterprise SaaS platforms.- Implementing robust transformation logic using Python, PySpark, SQL, and Java across structured and semi-structured datasets.- Ensuring data integrity, lineage, and performance across ingestion, transformation, and delivery layers.Requirements:- Data Engineering Experience: Substantial background (6+ years) as Data Engineer in enterprise-scale data infrastructures.- Leadership Skills: Proven leadership in technical project management and personnel mentorship.- Architectural Guidance: Track record of guiding architectural conversations in high-throughput data ecosystems.- Programming Mastery: Advanced proficiency in SQL and Python programming, with expertise in PySpark or Java.- Data Architecture: Comprehensive knowledge of data modeling, transformation methodologies, and data warehouse architectures.- Hadoop Ecosystem: Proven expertise with Hadoop ecosystem technologies.- Pipeline Development: Successful implementation of resilient, high-performance data pipelines meeting strict service level agreements.- Integration Expertise: Deep understanding of API integration, performance optimization, and data schema adaptation.- Language Proficiency: Advanced level of English.Additional skills preferred:- Workflow Tools: Practical implementation experience with Apache NiFi and Apache Airflow.- Development Environment: Expertise in containerized environments, developer tooling, and API frameworks.- AI/ML Platforms: Demonstrated capabilities in AI/ML platform support and generative AI pipeline development.- Innovation Contributions: Proven contributions to internal technological innovations within data engineering workflows.What to expect from us:- Competitive Compensation: Excellent payment in USD or your preferred local currency.- Paid Leave: Parental leave, vacation, and national holidays.- Dynamic Work Culture: Innovative and multicultural environment.- Elite Collaboration: Work with the global top 1% of talent.- Growth and Support: Mentorship and career development opportunities.If you are interested in being part of a team composed of the best professionals and working 100% goal-oriented in an innovative environment, do not hesitate to apply